Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost

ABSTRACT

A switch, switched architecture and process for transferring data through an FCAL switch is disclosed. The switch uses multiple switch control circuits each coupled to one FCAL network and all connected to a crossbar switch. The switch control circuits are coupled together by a protocol bus for coordination purposes. Local conversations can occur on each FCAL loop and crossing conversations through the switch can occur concurrently. The OPN primitive is used to establish the connection before any data is transferred thereby eliminating the need for buffer memory in the switch control circuits. The destination address of each OPN is used to address a lookup table in each switch control circuit to determine if the destination node is local. If not, the destination is looked up and a connection request made on the protocol bus. If the remote port is not busy, it sends a reply which causes both ports to establish a data path through the backplane crossbar switch.

[0001] This is a continuation in part of a co-pending patent applicationentitled FIBRE CHANNEL LEARNING BRIDGE, LEARNING HALF BRIDGE, ANDPROTOCOL, Ser. No. 08/786,891, filed Jan. 23, 1997, which is herebyincorporated by reference.

FIELD OF USE

[0002] Fibre Channel networks are known loop configuration networks thathave a plurality of known type nodes such as servers, printers, diskarrays etc. all connected together by the loop. Such networks use aunique protocol involving a plurality of 40 bit primitives that are usedto arbitrate for loop control, to establish connections and to carry outflow control for data transfers of frames of data. The flow controlinherent to the Fibre Channel Arbitrated Loop network (hereafter FCALnets) protocol has the advantage of eliminating the need for the nodesto have extensive buffering capabilities since the destination nodecontrols the amount of data it receives by transmission of an RRDYprimitive to the source node each time the destination node is ready toreceive another frame.

[0003] Fibre Channel networks emerged as a family of interconnectiontopologies to increase bandwidth over fast-wide SCSI networks and toincrease the number of server and storage elements that can be connectedto 126 over the 16 device limit of SCSI. Advantages of FCAL include thatdevices may be farther apart (up to 10 km) and more numerous and thatthe size of data transfers is very large compared to the overhead thatis required to set up every transfer. This makes FCAL very efficient andmore attractive than less efficient protocols such as TCP/IP overEthernet and SCSI over a bus connection.

[0004] Hub based network topologies are generally desirable because theyovercome certain limitations on the number of nodes that can be coupledto a network by breaking it up into segments coupled by the hub. ManyEthernet networks use hubs as do token ring networks. Hubs in FCALnetworks receive packets from a source node on an input line coupled tothe source node and rebroadcast the packet on an output line coupled tothe next node which rebroadcasts the packet to the next node and so on.The rebroadcast by subsequent nodes in the chain wastes computingresources. Switched topologies work differently in that packets are notrebroadcast, but instead are connected directly to the line coupled tothe destination node thereby eliminating processing by other nodes whichare not the destination to receive and rebroadcast messages not destinedfor that node.

[0005] Despite their advantages, a significant problem in FCAL networksis delay and this delay increases as the network scales up in size. Eachmeter of cable contributes 5 ns of delay. Further, each node contains anelasticity buffer or FIFO to absorb the differences between incoming andoutgoing data rates. Data passing through a node enroute to itsdestination passes through the nodes elasticity buffer and suffers atypical delay of 3 words. Typically, disk clusters are 10 drives to acluster with each drive being one node and imposing its own delay. Ifthere are 10 clusters coupled to a server, this would representtypically 5.3 microseconds of delay in transition of each primitive anddata frame travelling around the loop. In other words, this delay isimposed on each loop tenancy. In an I/O operation, there are typically 4tenancies for a write to disk, each involving 3 “round trips”: ARB,OPN-RRDY and Data/CLS (see ANSI standard X3T10 FCP which is herebyincorporated by reference). Thus, 12 delays would be suffered by eachcommand transaction. On a 100 node loop, this translates toapproximately 64 microseconds of delay per command.

[0006] The command overhead of modern disk drives is around 200microseconds and falling. The delay per command coupled to the commandoverhead of the drive imposes a significant penaly on performance ofapproximately 32%. For random access benchmarks with small I/O payloadstypical of database queries, the performance penalty becomes morepronounced. The problem manifests itself as the inability of the serverto achieve more I/O operations per second, regardless of how many moredisk drives are added to the system.

[0007] Spatial reuse provided by switches or hubs which allow concurrentloop tenancies is one way of reducing the delay problem. The IBM serialstorage architecture in the prior art is one method of providing spatialreuse.

[0008] Connection oriented switched topologies were tried in early FibreChannel Fabric networks to attempt to overcome the delay problems ofloops by cutting down the number of nodes each primitive and data framepasses through in getting from source to destination and providingspatial reuse. These early fabric switches were complicated, expensiveand slow, all of these characteristics being found quite undesirable byartisans of FCAL networks. In the early FC Fabric switches, an entireframe of data with a header that indicated the destination node to whichthe frame was directed was sent to the switch for purposes of requestinga connection. These early switch designs had microprocessors which wereused to implement several layers of software architecture to receive theframe, pass it up through various layers of processing to find the frameboundaries, crack the frame open, determine its destination address andthen attempt to find the destination node and make the switchingconnection. The entire frame of data had to be buffered during thisprocess of attempting to find the destination and make the properconnection. It was possible in this early design that the connection wasnever made, because, for example, the destination node was busy withanother conversation. The switch would then have to send a message backto the source that no connection was made and to try again later.Because of limited buffer space in the switch, the data in the originalframe might need to be overwritten by other data from a frame of dataembodying another request. In such a case, the switch would have to sendanother message to the source saying, “Sorry, I lost your data. Executeerror recovery protocol.” Error recovery protocols further complicatedthe operation and contruction of such systems. If a connection is made,the switch receives another frame of data back from the destination.This frame also must be received, have its boundaries detected and mustbe cracked open to examine its contents to see if the destination issaying, “Yes, I am available for a connection.” This type of switchproved to be unworkable and FCAL loops became the standard interconnectfor disks and servers.

[0009] Prior art Fibre Channel switches are commercially available fromAncor and Brocade Communications which provide spatial reuse andefficient link utilization. The FL_ports connected to these switchesalso address physical delays as they pertain to FCAL. However, theseswitches require link rate frame buffering to accomplish theirperformance levels, and also operate on the entire 24-bit addresscontained in the FC frame. In contrast, the invention described hereinuses zero buffering and an 8-bit address decode for a much moreefficient and inexpensive design.

[0010] Many network switched topologies that use entire frames of datato request a connection through the switch suffer these same drawbacks.The need for errory recovery protocols arise because of the potentialfor lost data arising from the fact that only limited amount of memorycan be put in the switch at realistic costs, and in heavy trafficsituations, the memory may be exhausted and some portion thereof mayhave to be rewritten with new data before the original data isdelivered. Memory is expensive, takes up space and complicates thedesign.

[0011] Examples of other network topologies other than Fibre ChannelFabric that suffer these same drawbacks are the 1 Gigabit Ethernet® andATM protocol networks now in public use.

[0012] The Fibre Channel Arbitrated Loop (FCAL) topology emerged as away of providing simple, low-cost connectivity to more nodes over ashared media than could be provided in point-to-point topologies withoutthe requirement for an expensive fabric switch. FCAL networks allow upto 126 node ports to be coupled by a shared media using a simpleprotocol without the need for a separate fabric switch. Unlike theswitched fabric topology which has a centralized approach to routing,FCAL networks distribute the routing function to each loop port. Thisreduces the cost of achieving interconnection since the loopfunctionality represents a relatively small addition to the normal portfunctionality that has to be present anyway. However, FCAL networkssuffer the disadvantage that the number of concurrent interconnectionspossible is smaller than in switched fabric networks because FCALnetworks are fully blocking topologies such that only two pairs of nodeson the loop can communicate at any particular time. All other nodes haveto wait until the first pair are done communicating before the loop isavailable for another pair to communicate. The address space is alsolimited to 126 nodes. Another problem with FCAL topologies is thattraffic originating from a source node had to travel through eachintermediary node on the loop portion between the source node and adestination node. Since each node imposed a delay, the overall bandwidthwas decreased since each loop tenancy involved a protocol wherein OPN,RRDY and CLS primitives and data frames had to travel through all theseintermediary nodes in order to complete the loop tenancy. Since no otherpair of nodes could communicate until the loop tenancy was complete, thedelays in transmission imposed by each node on each portion of theprotocol decreased overall bandwidth and throughput.

[0013] In an attempt to further increase bandwidth and concurrencywithout the extremely high cost of fabric switches, combinations of FCALloops with smaller fabric switches have been devised. This allows thecost per port of the fabric switch to be amortized over the the totalnumber of ports including those coupled to the subloops coupled to theswitch.

[0014] The problem with the approach of coupling multiple FCAL loopstogether by a fabric switch is that each FCAL subloop must be coupled tothe fabric switch by a complicated port called an FL_port. These areports which must be able to understand the FCAL loop protocol on oneside of the port and interface it with the very different andsubstantially more complex fabric switch packet switching protocol onthe other side of the port while also having bridging functionality.Such FL_ports must have protocol layers that understand each protocoland can do packet routing and communicate with each other.

[0015] Further, FL_ports are expensive to build. This is because of theinordinate amount of buffer memory that is needed in the front end ofthe FL_port to reconstruct the sequences of packets contained in oneFCAL tenancy. Modern day connectionless fabric switch protocols are purepacket switching while FCAL loop tenancy protocols are singleconversations which tend to be like a simple switched circuit andcontain many packets. That is, the FCAL loop tenancy protocol startswith an OPN primitive directed to a destination node which responds withan RRDY directed to the source. These primitives set up switches in thenodes so that the source and destination nodes talk directly to eachother through the loop segment between them and the intervening nodes bysending one or more frames of data until a CLS primitive is sent whichends the conversation.

[0016] In contrast, the packet switching done by an FL_port on theswitch backplane side is not nearly this simple. The packet switchingprotocol requires the FCAL frames output by a source node to be treatedas many individual packets. Those packets must be routed to the correctdestination FL_port and reassembled there without loss of data andlaunched on the FCAL loop on which the destination port is resident.Thus, FL_ports will receive packets from the fabric switch side and dataframes and primitives from the FCAL loop side and must have the memoryand intelligence to convert between the two protocols.

[0017] In the treatise Kemble, Arbitrated Loop, Chap. 1, pp. 18-19, FIG.12, Published by Connectivity Solutions, Tucson, Ariz. (1996), ISBN0-931836-82-4, Kemble proposes a “Smart Hub” conceptual network. Thisnetwork is comprised of a plurality of FCAL loops coupled together by asmart hub which has the intelligence to provide independent operationswithin each loop. When a source node want to exchange data with adestination node, it arbitrates for its local loop and attempts toestablish a loop connection with the destination. If the destination islocal, the smart hub simply acts as a repeater. If the destination portis not local, the smart hub intercepts the attempt to establish a loopconnection, acquires access to the proper destination loop andestablishes the loop connection between the source and destination loopsacting like a bridge. Thus one loop can talk to another withoutaffecting activity which is purely local to the other loops not involvedin the connection, but the other loops cannot set up simultaneousconnections to nodes on other loops during the tenancy across the smarthub of the first cross-boundary connection between the two loops alreadyconnected across the smart hub. This type arrangement cannot satisfy theneed for a fast switch which is affordable and provides the ability formultiple pairs of ports to communicate simultaneously across the switch.

[0018] As a response to the limitations on concurrency present inconventional FCAL topologies, the assignees of the present inventiondevised a network topology using intelligent hubs each of which hadrouting intelligence and each of which had its own subloop coupled to aplurality of conventional L_port nodes of a type used in conventionalFCAL topologies. Each hub was coupled to each other hub by a broadcastdata path and a return data path. By watching the addresses in the OPNprimitives and the flow of primitives, the hubs were able to deduce thelocation of the source and destination nodes and cut out all subloopsand nodes thereon that were not necessary for communication between thesource and destination nodes thereby decreasing unnecessary delay incompleting each loop tenancy and increasing bandwidth. Further, someconcurrency was supported in that tenancies between source anddestination nodes on the same subloop could be completed simultaneouslyon each subloop, thereby further increasing throughput. This technologyis described in U.S. Pat. No. 5,751,715 which is hereby incorporated byreference.

[0019] The assignee of the invention has also filed a co-pending patentapplication on a learning bridge for FCAL topologies such that two ormore FCAL loops can be coupled together by bridges. The bridges have theintelligence to examine the destination addresses of OPN primitivesreceived from their local loops and watch the primitives of looptenancies and the loops from which they came and to learn the locationsof various nodes on the loops to which they are coupled. Each bridgethen forwards OPN primitives from one loop to another if the destinationnode is on a different loop than the source node, but keeps the OPNlocal to the same loop as the source is on using a local bypass datapath if the destination and source nodes are on the same loop. Thisprovides the ability for concurrent loop tenancies to be occurring onthe two loops coupled to the bridge, although only one loop tenancy perloop is allowed at any particular time. This technology is described inthe parent application incorporated by reference herein.

[0020] Despite these improvements over standard FCAL topologies providedby the assignee, there is still a need for further improvements inconcurrency without the increased cost and increased protocol complexityof fabric switches. What is needed is a way to achieve the highconcurrency and bandwidth of fabric switch topologies without the highcost thereof. Therefore, a need has arisen for a relatively simple, FCALswitch which has little or no buffer memory needed therein and which iscapable of establishing connections very fast. Fundamentally, what theprior art is missing is a switch which can couple multiple FCAL loopstogether with high concurrency and the speed of a fabric switch thatuses N_ports and does not couple FCAL loops together without theexpense, complexity, memory demands and slowness of fabric switches thatuse FL_ports (FL_ports can support fast switching, but the amount ofmemory needed makes the cost prohibitive). This switch, in an idealworld, would have multiple ports, each of which is coupled to an FCALloop or an NL node and provide concurrency such that each port can talkto any other port at any time the other port is not already tied up inanother loop tenancy.

SUMMARY OF THE INVENTION

[0021] Two important attributes of all species within genus of theinvention are: first, the use of the destination address in an FCAL OPNprimitive (hereafter referred to as an OPN) instead of a frame header ofa frame of data to find the destination node and establish theconnection through the switch; and, second, using the normal flowcontrol primitives of the FCAL protocol for hold back purposes toeliminate the need for large buffer memories in the switch and so as toimplement a switching protocol to stream complete data frames fromsource to destination without storing any data frames in the switch, andwithout any packetizing, and without any segmentation and reassemblyprocessing, and without any error recovery protocols to retransmitdropped frames. Any species that shares these two characteristics iswithin the preferred genus of the invention. This preferred genusqualifies for class 2 Fibre Channel operation where frames cannot bedropped because the flow control nature of the switch prevents framesfrom ever being dropped because they are only transmitted when thedestination node has indicated it can receive them. The preferred genuscan also be operated in class 3 Fibre Channel operation where it ispermissible to drop frames and upper level protocols do error recoveryfor dropped frames even though the switch never drops frames.

[0022] A separate second genus of FCAL switches, suitable for Class 3Fibre Channel operation only, still uses the destination address in theOPN to find the remote port but uses buffers instead of hold back flowcontrol to complete the transaction to busy remote ports. Specifically,species within this genus will use the destination address of the OPNfrom the source node to find the location of the remote port. Then thestatus of that port will be checked. If the status is available, aconnection request will cause a connection to be set up between thesource node and the destination node via a source port connected to thesource node and a destination port connected to the destination node.The buffer comes into play when the destination port is busy. In thissituation, in the first genus described above, the normal primitives ofthe FCAL protocol are used for flow control to prevent the source nodefrom transmitting any frames of data until the destination port becomesavailable. In the second genus defined in this paragraph, a buffer bigenough to hold one or more complete frames of data is included in thefront end of each switch chip, or multiple buffers each big enough tostore a frame of data are included with each switch chip front end. Eachof these buffers will serve as an auxiliary switch port and have its ownconnection to the backplane in some species or a single sharedconnection to the backplane through a multiplexer can be used. Thepreferred species uses multiple buffers each with its own connection tothe backplane in addition to a connection directly from the switch portto the backplane for direct connections without buffering. In somespecies, a single shared buffer or multiple shared buffers on thebackplane or in some central location may be used.

[0023] In this second genus, the way the buffers are used is for thesource port to generate an RRDY sua sponte when it finds from a check ofthe scoreboard that the destination port is busy. The RRDY is sent tothe source node and causes it to output a frame of data. This frame ofdata is stored in the switch port's buffer. Then a message is sent tothe destination port indicating that the auxiliary buffer of the switchport is holding a frame of data for the destination port. This auxiliarybuffer ID is added to the camp list for the destination port. When thedestination port becomes available, a message is sent back on theprotocol bus indicating that the destination port is now available andnaming the backplane channel to use. A connection through the backplaneis then established to this channel by the auxiliary buffer connectioncircuitry and the destination port, and the data in the auxiliary bufferis transmitted. If the switch port has multiple auxiliary buffers, theyeach have their own IDs and, preferably, each has its own switchingcircuitry to make a connection to the backplane.

[0024] In this second genus, each auxiliary buffer has circuitry coupledto the return path to recognize RRDYs transmitted back by thedestination node and to count them (or store them) and to wait for aconnection between the source port and the RRDY counting circuit if theconnection is not continuous such as in some cases where multiplebuffers are present in each switch port. These stored RRDYs (or selfgenerated in the case of a count only) can be transmitted to the sourcenode in the case of full duplex or mixed with frames from a third nodein the case of a dual simplex connection and transmitted to the sourcenode. Each source port also has shared circuitry for each FCAL net whichrecognizes incoming RRDYs from the source node and counts them or storesthem. These source node generate RRDYs can be transmitted to thedestination node in the case of full duplex or transmitted to a thirdnode in the case of dual simplex.

[0025] Returning to consideration of the first genus, the normal bufferby buffer accounting and the hold back, handshaking nature of the FCALloop protocol with large data frames makes this genus of switchespossible and also very efficient.

[0026] The FCAL OPN primitive is a small 40 bit quantity which includesa code indicating it is an OPN primitive and includes a destinationaddress, and an optional source address if the OPN is full duplex.Receipt of the OPN starts the process carried out by the switch offinding the destination and causes establishment of the connection or anotification to the source that the connection could not be establishedbefore any data frame is ever transmitted to the switch. This lack oftransmission of any large data frame before establishment of theconnection means that the switching circuits connected to each FCAL loopcoupled to the overall switch structure do not need to have buffermemories to store the data while the connection is being made or thefact that a connection is not possible is established. This allows forgreatly simplified hardware. This means lower costs and greater densityof ports per chip. This cost advantage is a significant improvement overprior art approaches.

[0027] The major subclass of embodiments taught herein uses a pluralityof ports for connection to individual FCAL loops, and a crossbar switchwhich couples the ports together and which can implement any number ofseparate data transfer channels under control of the ports with the portusing the destination address information in the OPN primitives todetermine whether or not a connection through the backplane from oneport to another is needed. In the preferred species, the ports areintegrated circuits with many ports on one chip and a portion of adistributed crossbar switch also integrated on the chip to selectivelycouple the integrated circuit to one of the backplane data pathchannels. Each port is essentially a learning bridge front end with aninterface to the crossbar switch on the backend.

[0028] Fairness is provided, in the preferred embodiment, by a fairnesstoken which circulates to all the ports and which, when held by aparticular port, gives that port “high priority status”. This means thatif an OPN comes in to a port with the fairness token in its possessionand the destination node is on a remote port, the high priority statusof that port means that it can “camp” on the remote port and wait for itto be available and it is guaranteed access to the destination node nomatter how busy it is. Since the fairness token circulates, no port willever be starved from communication with a busy node.

[0029] Different variations or species within the subclass are taught.Distinctions between species within the subclass are based upon: the waythe destination node is found; the way in which the first port coupledto the source node signals the second, remote port that there is trafficwaiting for one of the NL nodes to which it is coupled; whether thecrossbar switch is central or distributed; whether the complete routingtable is stored in each port or there is a single separate routingtable, or whether there are partial routing tables stored in each port;whether a scoreboard is used or not to determine the status of a remotenode; and, if a scoreboard is used, whether it is distributed with acopy in each port or centralized and shared by all ports. All thesevariations between species and combinations of variations are equivalentto each other even though each has its own peculiar advantages anddisadvantages.

[0030] As an example of variations between species within the inventivegenus defined above consider the following. Location of the destinationnode can be by any of several means since the OPN includes thedestination address therein. In one embodiment, the destination addressfrom the OPN is use to address a lookup table which outputs data as towhich loop the destination node is on and to which switch chip or portcoupled to the destination loop the switching connection should be made.This embodiment has the advantage that all the connection information isimmediately available. This allows the connection to occur more rapidly.The disadvantage of this species is that the look up table is larger andeach port must carry a full copy of the routing table.

[0031] One alternative embodiment uses a destination location processwherein the destination address of the OPN from the source node is usedto address a lookup table (hereafter LUT) which only outputs a singlebit indicating, in one logic state, that the destination node is“local”, i.e., on the same FCAL network as the source node, orindicating, in the opposite logic state, that the destination node isnot on the local loop. If the destination is not local, the destinationaddress is broadcast as a location request to the other switch chipscoupled to the other FCAL networks connected to the switch. Each of theother switch chips then checks its local LUT using the destinationaddress to determine if it has the destination node on its loop. Theswitch chip that has the destination node finds this out from datareturned from its local LUT and then sends a message to the switch chipcoupled to the loop having the source node telling it to where theconnection is to be made and whether the connection can be made, i.e.,the loop upon which the destination node is not busy in anotherconversation and is available for the connection. The advantage of thisspecies is a smaller routing lookup table may be used in each port. Thedisadvantage is the requirement of more message traffic between chipsresulting in slower response.

[0032] An example of an FCAL switch within the genus of the invention isa bufferless switch for coupling to a plurality of FCAL nets and havinga crossbar switch and FCAL loop interface port circuits structured touse the OPN and RRDY primitives of the FCAL protocol for hold back flowcontrol to eliminate the need for a buffer with the ports and crossbarswitch structured to provide multiple simultaneous loop tenancies.

[0033] One embodiment for a protocol within the genus of protocols whichdefine the rules to set up a connection through an FCAL switch withinthe genus of the invention between a source node and a destination nodeand transfer data therebetween is:

[0034] 1) in a source node, arbitrating for and winning control of afirst FCAL net and transmitting an OPN primitive thereon, the OPNprimitive having a destination address of a destination node therein;

[0035] 2) receiving and latching at a first port of an FCAL switch theOPN primitive from the source node coupled to the first port by thefirst FCAL net;

[0036] 3) using the destination address in the OPN primitive as a searchkey to search a routing table to find the location of a destination nodehaving the destination address in the OPN or the ID of a port coupled byan FCAL net to the destination node, or both, and, if the destinationnode is coupled to the first port, passing the OPN primitive to thedestination node via the first port via a local bypass data pathcoupling an input of the first port to an output of the first portcoupled to the first FCAL net, but, if the destination node is coupledto a second port other than the first port, controlling a crossbarswitch to establish a data path between the first and second ports anddetermining if the second port is available, and, if so, sending the OPNprimitive to the second port indicating traffic is waiting to be sent tosaid destination node and latching the OPN in said second port;

[0037] 4) in the second node, arbitrating for control of a second FCALnet coupled to the second port;

[0038] 5) when control of the second FCAL net is won following saidarbitration, forwarding the OPN to the destination node;

[0039] 6) receiving an RRDY primitive or a CLS primitive from thedestination node in the second port and transmitting the primitive soreceived to the source node through a connection established acrosscrossbar switch, and through the first port and the first FCAL net; and

[0040] 7) for each RRDY received by the source node, transmitting aframe of data to the destination node through the first FCAL net, thefirst port, the data path through the crossbar switch, the second portand the second FCAL net without ever storing it in a buffer in theswitch, and continuing to pass data frames and primitives between thesource and destination nodes, until a CLS primitive is transmitted byeither the source node or the destination node, and then closing thedata path through the crossbar switch and relinquishing control of thefirst and second FCAL nets.

[0041] The preferred subclass of the switch utilizes the concepts of thelearning bridge taught in the parent application incorporated byreference herein for front end circuitry coupled to the FCAL net withbackend circuitry which is coupled to a crossbar switch. The bridgingfront end uses the destination address in the OPN to decide whether ornot to connect the front end circuitry to the back end circuitry. Thecrossbar switch implements a plurality of completely separate data pathsthrough the switch each of which can couple two port together. Theprovision of multiple separate data paths through the crossbar switcheliminates any bottlenecks which could occur if a multiplexed data buswere to be substituted for the crossbar switch. It is within the genusof the invention however to substitute a multiplexed bus for thecrossbar switch using any form of multiplexing.

[0042] Thus, the switch apparatus genus could be generally described asincluding multiples species, each comprised of a plurality of halfbridges, each with a front end for connnecting to an FCAL loop and abackend coupled to either a crossbar switch or a multplexed bus alongwith suitable control circuitry to use the destination addresses in OPNprimitives to determine whether a connection between two ports throughthe crossbar switch or multiplexed bus is necessary and, if necessary,for establishing the connection.

[0043] In the preferred embodiment, each half bridge is one port. In thepreferred embodiment, the half bridges are implemented as integratedcircuits with a multiplicity of half bridges on every chip with eachhalf bridge building its own routing table by a passive learningprocess. An alternative embodiment uses an active discovery process tobuild the routing table.

[0044] The switch architecture can be thought of as a multi-port switchwith a stack of learning half bridges substituted for each FL_port of aprior art fabric switch, with each half bridge on each layer beingcoupled to its own local FCAL loop or single NL node. The other side ofeach half bridge is connected to the high speed crossbar switch in thepreferred embodiment so that it can be connected to the other halfbridges. The crossbar switch can be though of a stack of separate layersof separate high speed backplane data paths connecting all the halfbridges together by way of a switching network between the high speedbackplane data path layers. The switching network functions to establishselective connections between layers and can be controlled such that anybridge on any layer can talk to any other bridge on any other layer.This allows multiple concurrent connections across the switch between aplurality of pairs of source nodes on one loop and a plurality of pairsof destination nodes on other loops or source and destination nodescoupled individually to half bridges. The switch architecture allowssimultaneous purely local loop tenancies on any FCAL net coupled to anyparticular half bridge so long as another node on the FCAL net is notinvolved in a loop tenancy which involves communication across theswitch from one port on one FCAL net to another port on another FCALnet.

[0045] Flow control using the OPN primitive only to establishconnections across the switch is used to eliminate the need for largeamounts of memory. As a result, the switch is capable of operating at ahigh throughput rate, but neither the half bridges nor the crossbarswitches has the amount of memory of an FL_port of a fabric switch thatwould be required to make the fabric switch capable of operating at thesame throughput rate.

[0046] Another significant advantage of the invention is that the nodeson the individual FCAL nets can be conventional NL node designs whichalready exist. An NL node is a node on an FCAL net which understands andcan implement the FCAL flow controlled loop connection protocol betweensource and destination nodes to transfer data using OPN, RRDY and CLSprimitives and large data frames. Because the embodiments of theswitches described herein are all compatible with conventional NL nodes,the genus of switches described herein has the advantage that when thenetwork is upgraded, only the switch needs to be upgraded and all thenodes can remain the same thereby saving substantial expense to thecustomer.

[0047] Dual simplex capability is also taught to improve the throughputof any network of FCAL nets coupled by any type of switch. Dual simplexcapability allows a source node on a first FCAL net which istransmitting data on a front channel connection to a destination node ona second FCAL net through a switch connection to simultaneously receivedata via a back channel connection from a third node on a third FCALnet. This is advantageous to improve throughput because in many cases,destination nodes to which data has been transmitted have no data totransmit back to the source node that sent them the data while othernodes do have data to be transmitted to the source node.

[0048] Dual simplex capability is accomplished in all species within thegenus of the invention to further increase throughput. It isaccomplished by three basic steps, illustrated in FIG. 15:

[0049] 1) establishing a front channel half duplex data path between asource node and destination node on different FCAL nets (step 350) andstripping and storing or counting any buffer credit RRDY primitivesoutput by the source node and not transmitting them to the destinationnode (step 352);

[0050] 2) establishing a back channel data path between a third node andsaid source node but not transmitting to said source node any OPNprimitive emitted by said third node (step 354), and transmitting anumber of RRDYs either equal to the number of RRDYs output by saidsource node or the number of RRDYs needed by said third node to send allthe data it has to said source node before closing said back channelconnection (step 356), transmission of said RRDYs being one at atime—any excess RRDYs not used by the third node are saved for use byanother third node in a subsequent dual simplex back channel connection;and

[0051] 3) receiving any RRDYs transmitted by said destination node andmixing them in with data frames and/or primitives transmitted on theback channel by the third node so as to exercise flow control ontransmissions of data frames from the source node to said destinationnode (step 358).

BRIEF DESCRIPTION OF THE DRAWINGS

[0052]FIG. 1 is a drawing of one example of an FCAL switchedarchitecture according to the teachings of the invention.

[0053]FIG. 2 is a drawing of another example of an FCAL switchedarchitecture according to the teachings of the invention having multiplesubloops within some of the FCAL coupled to the switch.

[0054]FIG. 3 is a block diagram of the preferred switched FCALarchitecture.

[0055]FIG. 4 is a flow chart of the processing which occurs for a purelylocal transaction.

[0056]FIG. 4 is a block diagram of the general preferred architecture ofa switch to couple a plurality of FCAL nets to provide spatial reuse.

[0057]FIG. 5 is a block diagram of the specific preferred “switch slice”architecture of a switch to couple a plurality of FCAL nets to providespatial reuse with multiple port circuits and a portion of the crossbarswitch integrated on each switch chip.

[0058]FIGS. 6A through 6C are a flow diagram of a routing algorithm forgeneral switch mode operation (non dual simplex).

[0059]FIG. 7 is a block diagram of the preferred, architecture of eachswitch chip in the FCAL switch system.

[0060]FIG. 8 is a diagram of an FCAL switch with two switch chipsconfigured to run in hub mode.

[0061]FIG. 9 is a diagram of an FCAL switch system with two switch chipsconfigured to run in switch mode and illustrating loop-local, chip-localand remote port simultaneous loop tenancies.

[0062]FIG. 10 is a block diagram of the multiplexer structure in eachswitch chip that allows port bypass mode, parallel loopback mode andserial loopback mode to be implemented.

[0063]FIG. 14 is a table describing and naming each state in the loopport state machines of each port of each switch chip.

[0064]FIG. 12 is a table of the source port fill word generation forvarious input words and states.

[0065]FIG. 13 is a table of the destination port fill word generationfor various input words and states.

[0066]FIGS. 14A through 14E are diagrams of the five different messageformats on the protocol bus.

[0067]FIG. 15 is a generic flow diagram illustrating the minimum basicsteps each species in the genus of the invention would have to carry outto implement dual simplex operation.

[0068]FIGS. 16A through 16D are a flow chart of the specific stepscarried out by the preferred embodiment to carry out dual simplexcommunications.

DETAILED DESCRIPTION OF THE PREFERRED AND ALTERNATIVE EMBODIMENTS

[0069] The published Fibre Channel Arbitrated Loop standardsmemorialized in the following ANSI standards are hereby incorporated byreference: X3.230-1994 describing the physical and signalling interface;X3.297-1996 describing the physical and signalling protocol; X3.272-1996describing the general FCAL protocol and TR-20-199X, T11 Project1235-DT, Fibre Channel Fabric Loop Attachment (FC-FLA).

[0070] Referring to FIG. 1, there is shown one embodiment of a switchedFCAL architecture. FCAL switch 10 is coupled to four FCAL networks(hereafter sometimes referred to as FCAL nets) 12, 14, 16 and 18 in thisexample. Each FCAL net can have one or more NL nodes thereon. Each ofthe four FCAL networks is coupled to a plurality of NL nodes which haveconventional structure and which can carry out FCAL arbitration, datatransfer and flow control operations on the FCAL networks. Each node isassigned an address from one of the 127 possible FCAL addresses.

[0071] In FIG. 1, each node is given a number symbolizing its addressdesignated in the figure by N and a number inside the circle symbolizingthe node.

[0072] The function of the switch 10 is to increase total throughput ofthe system by allowing concurrent conversations to be occurring betweenpairs of NL nodes, and by doing so in a manner that is not limited byany restriction against “crossing conversations” in the switch itself. Anew standard for FCAL topology proposed by IBM involves counterrotating,separate FCAL rings with nodes coupled to both counterrotating rings.This allows concurrent conversations to occur, but the conversationscannot “cross”, i.e., it is illegal in this protocol for bothconversations to require the same segment of an FCAL as part of the datapath for the conversation. “Conversation”, as that term is used herein,means a data transfer between two different nodes. The function of theswitch 10 according to the teachings of the invention is to allow asmany concurrent conversations as possible except that no two sourcenodes can be talking to different destination nodes on the same FCALnetwork. In other words, each of FCAL networks 12, 14, 16 and 18 islimited to only one conversation at a time even though data flow frommultiple conversations may be simultaneously be flowing through switch10. This is done by establishing “virtual channels” for eachconversation through the switch using separate data paths (ormultiplexing techniques in some embodiments). The physical configurationof the switch 10 is not currently believed to be important so long as ituses the destination address in each OPN from a source node to controlsetting up a separate data path through the switch for transfer of databetween that source node and a destination node and uses flow controlprimitives of the FCAL protocol to control the flow of data such thatthe switch does not need to have a buffer memory big enough to hold anentire FCAL frame.

[0073] The fact that only 127 addresses are possible is an inherentlimitation of the FCAL protocol but it is also an advantage in thefollowing way. Because there are only 127 possible addresses, nomicroprocessor is needed in the switch. With a manageable address space,the location of the destination node can be determined by looking up thedestination addresses using a state machine and a lookup table in eachswitch control circuit coupled to an FCAL network. The lack of amicroprocessor both makes the switch faster and cheaper.

[0074] In the topology of FIG. 1, each FCAL network 12, 14, 16 and 18has a one Gigabit/second data throughput capacity. Therefore, themaximum throughput of the system shown would be 4 Gigabits/second ifeach of the FCAL networks 12, 14, 16 and 18 had a purely localconversation occurring thereon.

[0075] One way that the FCAL switched architecture according to theteachings of the invention can increase throughput is to allow multiplelocal conversations to occur on each FCAL network through use ofbridges. An example of a topology that can take advantage of thisfeature is shown in FIG. 2. There, switch 10 is coupled to FCAL networks12, 14, 16 and 18. However, FCAL network 12 is divided into two FCALsubnetworks 12 and 20 by FCAL bridge 22, and FCAL network 14 is dividedinto two FCAL subnetworks 14 and 24 by FCAL bridge 26. Likewise, FCALnetwork 16 is divided into three FCAL subnetworks 16, 28 and 30 by FCALbridges 32 and 34. FCAL bridges 22, 26, 34 and 32 can have theconstruction detailed in co-pending U.S. patent application entitledFIBRE CHANNEL LEARNING BRIDGE, LEARNING HALF BRIDGE, AND PROTOCOL, Ser.No. 08/786,891, Filed Jan. 23, 1997, the contents of which are herebyincorporated by reference. The maximum throughput in the topology ofFIG. 2 would occur when each subloop was having a purely localconversation. In the topology of FIG. 2 with 8 subloops, the maximumthroughput would be 8 Gigabits/second.

[0076]FIG. 3 is a high level block diagram of the internals of the FCALswitch 10 for the preferred embodiment using a crossbar switch 44 tomake the switched electrical connections between FCAL networks 12, 14,16 and 18. Each of the FCAL networks 12, 14, 16 and 18 is coupled to itsown switch control circuit, i.e., switch control circuits 36, 38, 40 and42, respectively. The switch control circuits 36, 38, 40 and 42 aredistinguished from switch control circuits of prior art switch designsby the fact that none of them have buffer memory therein large enough tostore an entire frame of data or enough memory to carry out thepacketization process of prior art switch designs.

[0077] The function of the switch control circuits 36, 38, 40 and 42 isto transmit primitives and data involved in FCAL arbitration, datatransfer and flow control to the appropriate loop segment, do bypassswitching when a conversation is completely local so as to bypass thecrossbar switch 44, to collectively locate the destination nodes whenOPNs are received, and to send appropriate control signals to thecrossbar switch once the destination node has been located so as toconnect the appropriate FCAL networks together to complete theconversation.

[0078] The crossbar switch 44 has 4 inputs and 4 outputs with 1 inputand 1 output for each FCAL. The crossbar switch serves to make a datapath through the appropriate switch control circuit to the input andoutput of one FCAL having the source node thereon to the output andinput, respectively of another FCAL having the destination node thereonthrough the appropriate switch control circuit so as to provide achannel through which the conversation between the source node and thedestination node may proceed. The crossbar switch must be able tosimultaneously connect the input and output of another FCAL havinganother source node thereon to the output and input, respectively of yetanother FCAL having another destination node thereon to provide a datapath or channel for a second concurrent conversation. This concept isextended for as many pairs of FCAL as are connected to the switch. Theparticular connections that are made are controlled by enable signals oncontrol buses 46, 48, 50 and 52. These enable signals are generated bythe switch control circuits based upon the locations of the destinationnodes for the concurrent conversations. Any crossbar switch that canperform the above described function will suffice providing it cansupport the necessary data rate and traffic volume.

[0079] The switch control circuits 36, 38, 40 and 42 are coupled by aprotocol bus 54. In some embodiments, this protocol bus may bemultiplexed using any multiplexing scheme such as TDMA. In otherembodiments, crossbar switch 44 can be omitted and a TDMA bussubstituted with timeslot assignments taking the place of assignments ofparticular backplane channels through the crossbar switch and messageson the protocol bus updating all switch ports with information aboutwhich timeslots are in use and which timeslots are available.

[0080] Referring to FIG. 4, there is shown a block diagram of thegeneral architecture of the preferred Fibre Channel switch. The switchis comprised of a crossbar switch 100 coupled to a plurality of learninghalf bridges such as are represented typically by blocks 102, 104 and106. Each learning half bridge has a port having an input and an outputfor coupling to separate input and output wires or fibers of the FibreChannel link. The Fibre Channel link of each port can be coupled to anindividual NL port such as block 108 or an FCAL net such as isrepresented by NL ports 110 and 112 and links 114, 116 and 118. An FCALnet can have one or more NL nodes on it, so even the connection to thesingle node 108 is an FCAL net using a Fibre Channel protocol as thoseterms are used in the claims.

[0081] Each learning half bridge in the switch can have a similar frontend structure and mode of operation as that described in the parentapplication Ser. No. 08/786,891, filed Jan. 23, 1997 which isincorporated by reference herein, but preferably has the structuredescribed below in FIG. 7. The “front end” structure refers to thecircuitry that is coupled to the port and any Fibre Channel linkconnected thereto. Each learning bridge port circuit (hereaftersometimes referred to as a port) is coupled to a 50 Mhz 24 bit protocolbus 121. Link 123 represents this connection between port 106 and theprotocol bus. The protocol bus is where each port posts its transactionsindicating its local FCAL net is busy or has become available so as toupdate that port's status information in a local copy of scoreboardtable 125 in memory. In the preferred embodiment, each port maintains asynchronized local copy of the scoreboard table and the contents arewritten to the scoreboard by learning from messages posted on theprotocol bus. “Synchronized” means all copies of the scoreboard have thesame information at all times to avoid “fatal embrace” scenarios (fatalembraces are discussed in the parent bridge case).

[0082] In the alternative embodiment of FIG. 4, the scoreboard 125 androuting table 127 are shown as central shared circuits, but in thepreferred embodiment of FIG. 5, every port circuit 124, 126 and 128 hasits own copy of the scoreboard table and a routing table. The protocolbus is also the communication path used by a first port when an OPN to aremote destination node comes in to send a request for a connection to asecond port coupled to the destination node. The connection request issent on the protocol bus after the first port checks the scoreboard anddetermines that the second port is available. The request causes thesecond port to begin arbitrating for control of its local FCAL net. Thescoreboard contains circuitry to read messages on said protocol bus anduse the information therein regarding the identity of the sending portand the status codes in the message to update the status entry for thesending port. The structure of this circuitry is not critical, andanything that can perform this function will suffice.

[0083] Each learning half bridge includes a streaming back end whichdrives a bidirectional port such as ports 120 and 122 coupled to thecrossbar switch. This streaming backend allows large strings of data tobe sent all the way from the source node to the destination node as astream without the need to buffer any of it. The structure of thestreaming backend and the half bridge front end is such that hold backflow control is used so that no frames are stored—they just stream allthe way from the source to the destination under buffer by bufferaccounting using primitives for buffer management in the nodesthemselves and not in the switch. As a result, no frames are everdropped and there is no need for buffer management in the switch itselfnor any need for dropped frame error recovery protocols in the switch ornodes.

[0084] In sharp contrast, in the connectionless fabric switches of theprior art for both FCAL nets and other protocols, the conventionalwisdom is that the essence of the problem is in buffer management. It isactually possible in the prior art fabric switches in the FCAL netenvironment to drop frames because of conjection related to trafficproblems. When a frame is dropped, a complex error recovery protocolneeds to be executed in the nodes (node can mean computer, disk drive orother computing machine coupled to the network) involved to retransmitthe dropped frame, or the entire I/O transaction. The error recoveryprotocol is costly in terms of overhead, and many I/O applications onthe nodes are not designed to gracefully handle dropped frames therebycreating the possibility of errors and increasing the complexity of thenodes in requiring error recovery protocol software to exist in thenode. The acknowledged, connectionless class of service in Fibre Channel(Class 2) does not eliminate or reduce the frequency of dropped frames.the only advantage is that class 2 has negative acknowledgements to thesource node from the fabric or destination node if/when any frames aredropped. The only advantage is that this negative acknowledgement mayprovide more timely notification of dropped or reected frames. it doesnot improve the intrinsic reliability of FC networks.

[0085] In the invention, the hold back flow control which is part of theFCAL protocol is used to advantage to eliminate the need for buffermemory in the switch. Thus, in the invention, frames are not held inbuffer memory, so they cannot ever be dropped because of congestion. The“hold back flow control” used by the invention prevents data from everbeing transmitted from the source until the switch is ready to stream itto its destination. Therefore, there is no need for any complicatederror recovery protocols in the switch or in the nodes and no need formassive amounts of memory in the switch. This is only possible in theFibre Channel protocol where the possibility of using hold back flowcontrol exists by virtue of the use of primitives and buffer by bufferaccounting. There is no low level flow control in ATM or 1 GB Ethernetprotocols. Those protocols have X-on and X-off flow control on a higherlevel of the ISO level. This X-on and X-off flow control is “embedded”in the data characters, in contrast to Fibre Channel wherebuffer-to-buffer flow control is external to any data or frames. InFCAL, the OPN primitive signals when a source node has data to send, andthe destination node then signals with a RRDY primitive sent back to thesource saying it has reserved space to receive a frame. One and only oneframe is then sent and no other frames are sent until the sourcereceives another RRDY primitive from the destination node. This use ofprimitives and reserved space and buffer by buffer accounting is themeaning of hold back flow control which is used in the invention.

[0086] The lack of buffer memory in the half bridges makes them highlycompact and susceptible to full integration since memory is a largeconsumer of die area. The ability to integrate multiple half bridges ona single integrated circuit die reduces the cost per port to asubstantially lower cost than for a corresponding number of FL_ports.

[0087] Prior art switch designs typically had a crossbar switch chipcoupled to a plurality of switch module chips each of which had asubstantial amount of the die area consumed by buffer memory cells. Thustwo separate chip designs were necessary to implement a switch. In thearchitecture of the invention since part of the crossbar switch and theport circuitry for several ports are on one IC die, only one chip designis necessary to implement a switch. The various portions of thedistributed crossbar switch on different chips, when coupled together,form one complete crossbar switch. This crossbar switch has 14 backplanechannels, but they are not all available because each switch port chipalso includes access circuitry to the backplane channels. If only twoswitch port chips having the architecture of FIG. 7 are coupledtogether, the amount of backplane access circuitry present is not enoughto access all 14 backplane channels. The number of backplane channelsthat can be accessed with the available access circuitry defines aconcept of “available bandwidth”. As the number of switch port chipscoupled together grows, the amount of available backplane accesscircuitry grows so that the number of backplane channels availablegrows, i.e., the available bandwidth grows which is as it should bebecause traffice volumen is increasing with an increasing number ofports. In other words, each switch port chip is a network slice and anetwork can be fabricated with as many ports as needed, and the crossbarswitch will automatically grow in available bandwidth in proportion tothe number of ports adequately to handle the increased traffic.

[0088]FIG. 5 illustrates the actual architecture of the preferredspecies of the separate channels backplane subclass of switches. Eachblock 124, 126 and 128 represents an integrated switch chip having aplurality of ports (learning half bridges) and a portion of the crossbarswitch thereon. The function of each of the half bridge ports is toprovide switching capability between networks connected using the FibreChannel physical layer and the Fibre Channel Arbitrated Looptransmission layer protocol. A point-to-point backplane data path 130comprised of a plurality of channels through the distributed crossbarswitch carries data from port to port. Each channel has a conductor inboth directions. Each channel is a separate one gigabit per second datapath which is selectively coupled by the crossbar switch to the twoports involved in a loop tenancy and is not shared by any other ports atthat time. In one example of an architecture in accordance with thepreferred embodiment, there are 14 switchable channels for a total of 28Gbits/sec of bandwidth. The function of the backplane data path 130combined with the crossbar switch is to carry multiple conversations ofdata between ports simultaneously.

[0089] Since each of the switch chips 124, 126 and 128 contains aplurality of independent ports, spatial reuse and concurrency isachieved in the architecture of the invention in three ways: first, anypurely local conversations between source and destination nodes on thesame FCAL net can simultaneously occur on all FCAL nets withoutconsuming either switch chip or backplane bandwidth; second, anyconversations between different FCAL nets coupled to the same switchchip can occur purely within each switch chip without consuming anybandwidth on the backplane bus; and, third, the multiple backplane datapaths allow multiple simultaneous conversations between nodes on FCALnets coupled to different switch chips. For a first example, with 36ports coupled to 36 FCAL nets, a peak achievable bandwidth of 72Gbits/sec is achievable with no backplane channel utilizationrepresenting 36 simultaneous local full-duplex conversations. The peakbandwidth through the switch is equal to 2 Gbits/sec per backplanechannel (1 GB/sec send and 1 GB/sec receive), plus an additional 2Gbits/sec for each purely local conversation that can be simultaneouslyserviced. Thus, with 14 backplane channels all in use and 36 portscoupled to 36 FCAL nets with all ports not coupled to one of the 14backplane channels having local conversations ongoing, the total trafficvolume is 28 GB/sec through the switch plus 22×2 GB/sec equalling 44GB/sec for a total of 28+44=72 GB/sec.

[0090] In an exemplary embodiment of an FCAL switch using the teachingsof the invention, each switch chip has 3 ports and there are 12 switchchips for a total of 36 ports in this embodiment. Each switch chipsupports 14 backplane channels such that up to 14 remote full-duplexconversations through the switch 136 can be supported by 28 portscoupled to the 14 backplane data paths plus 8 purely local full-duplexconversations between the remaining 8 ports for an aggregate peakbandwidth of 44 Gbits/sec. Port and backplane contention will reduce thetotal bandwidth below this peak, but average throughput and arbitrationlatency of a switched system will still be faster than an arbitratedFCAL net of the same size. Spatial reuse provided by a switch allowsapplications which make use of concurrent conversations to use FCALlocal loops and a switch to overcome the one loop tenancy at a timelimitation of pure Fibre Channel Arbitrated Loop networks without aswitch.

[0091] Each switch chip is coupled to a fairness token bus 99, which, inthe preferred embodiment, takes the form of a two wire bus. A fairnesstoken circulates to all ports on this bus enabling a “round robin”fairness algorithm. Each port has circuitry to receive the fairnesstoken, hold it for a short time and forward it to the next port so thatthe token reaches all ports eventually. Each port has a priority levelassigned to it that is used in arbitration within the switch of multipleaccess requests to a busy destination node. When the fairness token isin the possession of a port, that port has the highest priority leveland is guaranteed access. When a port has the highest priority level, itcan “camp” on a busy remote port, and be guaranteed that it will begiven access when the port is available. When access has been granted,the token is forwarded. This prevents starvation of any port fromcommunication with a busy node.

[0092] In the preferred embodiment, data path 130 is comprised ofmetallic data paths on the printed circuit board on which the port chipsare mounted couples the outputs of each crossbar switch portion of oneof the port chips to the input of the crossbar switch portion of anotherport chip. Each bridge portion has three inputs and three outputs (oneinput and output for each of the three half bridges on each chip) forFCAL net primitives and frame data. Each pair of one input and oneoutput is called a port and can be coupled to a single NL node or anFCAL net.

[0093] Note that the distributed nature of the crossbar switch and thefull integration of the ports with the crossbar switch makes thearchitecture of the switch modular and easily expandable. That is, thearchitecture represents a sliceable architecture which can be built intoas large a switch with as many ports as is necessary limited only by themaximum number of permissible addresses in the FCAL address space of 128nodes. Integration of the ports with a crossbar switch slice on the samechip allows switches to be built with as few as two switch chips or manymore because the crossbar switch is scaleable and because each chip canbe coupled by its portion of the crossbar switch to any one of theplurality of separate backplane channels. Prior art fabric switches hadseparate, nonscaleable circuits for the crossbar switch. This meant thatthe crossbar switch had to be built as complex as was necessary tohandle up to the maximum number of ports. If fewer than the maximumnumber of ports were used, a large part of the prior art crossbar switchcapacity was wasted. In contrast, in the invention, because of the fullintegration of all necessary circuitry onto one chip including a portionof the crossbar switch, the crossbar switch is scaleable, and all sizesof switches are easily constructed and quite affordable because of theinexpensive, fully integrated construction with no buffer memory and nocomplicated error recovery protocols.

[0094] Dual Simplex

[0095] FCAL NL nodes are capable of duplex communications in that theycan send data at the same time they are receiving data. The problem thatconventional FCAL nets and fabric FL port based switches fail to addressis the fact that typically the destination node in more highly populatedFCAL nets is less likely to have data for the source node but othernodes may have data for the source node which they cannot send since thesource node is tied up in a conversation with the destination node.Thus, there is a waste of bandwidth. Typically, node adapter cards thatinterface a computer or disk drive to the FCAL net cannot re-shuffletheir transmit queues in order to find a frame destined for the currentsource node and move it to the top of the transmit queue. Thus, head endblocking usually occurs and causes a full duplex connection to beeffectively simplex. In a 100 mbyte/sec link for example, NL nodes cansimultaneously burst out at 100 mbyte/sec while they are simultaneouslybursting in at the same rate. If the destination node has no data tosend to the source node, the 100 mbyte/sec of burst in bandwidth of thesource node is wasted in conventional fabric switches with FL ports.

[0096] In conventional FCAL nets, the protocol requires that nodes mustmust reserve all resources required for a duplex conversation even incases where there is only traffic in one direction. This means, in thecontext of FIG. 5, node 101 can be sending data to node 103 but node 103is required by the rules of the protocol to be sending data only to node101 during the same loop tenancy. This is true even though it may haveno data to send to node 101 and even though it may have data to send tonode 105. Likewise, if the switch of FIG. 5 followed the normal FCALprotocol, if node 138 had frames to send to node 103, node 103 would berestricted to sending data only to node 138 even if node 103 had no datafor node 138 but did have data for node 140.

[0097] This waste of bandwidth is remedied by the provision of dualsimplex conversations across the switch of the invention. The ports andcrossbar switch of the invention allow dual simplex conversations acrossthe switch but not within any local loop coupled to a port. Simply put,dual simplex capability allows nodes other than the destination node tosend data frames to the source node while that source node is sendingdata to its destination node. In the example of FIG. 5, a dual simplexconnection for two separate one-way conversations could occur asfollows. Suppose, node 138 has data to send to node 103 but node 103does not have any data for node 138. Suppose also, node 144 has data fornode 138. With the dual simplex capability of the invention, node 138can send its data to node 103 while simultaneously receiving data fromnode 144.

[0098] Dual simplex capability is accomplished in all species within thegenus of the invention by three basic steps, illustrated in FIG. 15:

[0099] 1) establishing a front channel half duplex data path between asource node and destination node on different FCAL nets (step 350) andstripping and storing or counting any buffer credit RRDY primitivesoutput by the source node and not transmitting them to the destinationnode (step 352);

[0100] 2) establishing a back channel data path between a third node andsaid source node if said source node is dual simplex capable but nottransmitting to said source node any OPN primitive emitted by said thirdnode (step 354), and transmitting a number of RRDYs either equal to thenumber of RRDYs output by said source node or the number of RRDYs neededby said third node to send all the data it has to said source nodebefore closing said back channel connection (step 356)—transmission ofsaid RRDYs being one at a time—any excess RRDYs not used by the thirdnode are saved for use by another third node in a subsequent dualsimplex back channel connection; and

[0101] 3) receiving any RRDYs transmitted by said destination node andmixing them in with data frames and/or primitives transmitted on theback channel by the third node so as to exercise flow control ontransmissions of data frames from the source node to said destinationnode (step 358).

[0102] In the preferred species of switch within the genus, thefollowing steps are carried out to implement dual simplex dataexchanges:

[0103] 1) in a port of the switch coupled to an FCAL net, receiving afull duplex OPN(Dest, Src) from a source node designating a destinationnode address, Dest, and a source node address, Src, and converting thefull duplex format OPN(Dest, Src) to a simplex or half duplex formatOPN(Dest, Dest) and establishing a “front channel” connection throughthe crossbar switch to the destination node (in the preferredembodiment, conversion from full to half occurs if and only if aconfiguration bit is set in each port to allow dual simplex, and, if setto allow it, always converts all full duplex OPNs to half duplex OPNseven if no dual simplex connections are expected—in alternativeembodiments, conversion from full to half duplex always occurs and isnot configurable and in other embodiments, dual simplex is neverallowed);

[0104] 2) storing at least the source node address in the port of theswitch coupled to the source node, hereafter referred to as the sourceport;

[0105] 3) stripping buffer credit RRDYs output by the source node andnot transmitting the RRDYs output by the source node to the destinationnode of the front channel connection, and either storing or counting theRRDYs output by the source node and making them available to a thirdnode to send data to the source node either by transmitting stored RRDYsoutput by the source node one at a time to the third node or byconveying the number of RRDYs output by the source node to a third portcoupled to the third node and synthesizing in the third node a number ofRRDYs equal to the value of the count and sending the RRDYs to the thirdnode one at a time (the preferred embodiment does not store the RRDYs—itjust counts them and regenerates them in the source port when needed fortransmission to a third node);

[0106] 4) updating status data in a “scoreboard” memory of each port toindicate which busy ports are coupled to source nodes of front channelconnections and thus are available to receive frames in dual simplexcommunication from third nodes—the scoreboard memory is a memory thatstores status data such as whether or not a port is busy, and, if busy,whether it is coupled to the source node of the conversation it isengaged in and is thus open to receive data from a third node in a dualsimplex communication;

[0107] 5) establishing a “back channel” connection through the crossbarswitch from the third node to the source node and transmitting an OPNfrom the third node to the source switch port—then the stored sourcenode address in the source port is compared to the destination nodeaddress of the OPN transmitted from the third node and allowing dualsimplex communication to proceed on the back channel if the addressesmatch, but blocking dual simplex communication with this particularthird node if the addresses do not match by sending a CLS primitive tothe third node;

[0108] 6) in the event dual simplex communication is allowed, deletingthe OPN from the third node and never transmitting it to the sourcenode, and releasing stored RRDYs to the third node (either one at a timeor all at once) thereby allowing the third node to start sending dataframes to the source node; and

[0109] 7) thereafter data frames are transmitted by the third node tothe source port with one data frame transmitted in response to each RRDYreceived by the third node—these data frames are mixed with RRDYsreceived from the destination node via the front channel connection—thecombined data frames and RRDYs are transmitted to the source node viathe back channel connection.

[0110] The “back channel” is the data path going into the source nodealong with a connection through the switch ports and backplane ifnecessary to couple the third node to the source node and is the channelused by a third node to send dual simplex data frames into the sourcenode.

[0111] The format conversion mentioned above converts the OPN(Dest, Src)received from the source node 138 into an OPN(Dest, Dest). Thisconversion occurs as the OPN is propagated through the source portbefore it is sent to the remote port coupled to the destination node.Conversion of the OPN primitive format to OPN(Dest, Dest) and takingaway the buffer credits from the destination node converts the normalfull duplex loop tenancy to a half duplex or simplex loop tenancy wheredata flows in only one direction from the source node to the destinationnode and the destination node cannot send any frames back to the source.

[0112] Borrowing the buffer credit transmitted by the source node andgiving it to another node that has data to send to the source node isimportant to dual simplex capability. In the specific example beingconsidered, it means that any RRDY primitives output by source node 138that would otherwise give destination node 103 one or more buffers ofcredit for return data are, instead, stripped by the source port beforethe OPN is transmitted to the remote port, and stored so that they maybe given to node 144 after a simplex back channel connection isestablished. Specifically, suppose source node 138 transmitted OPN(103,138) RRDY RRDY. Port 126 would convert this to OPN(103, 103) andtransmit it to switch port 128 when the connection through the backplanebetween switch ports 126 and 128 has been established in any of the waysdescribed herein. Port 126 would latch the two RRDYs or increment acounter of buffer credits to a count of two and give them to node 144one by one after a back channel connection was established between node144 and node 138.

[0113] Swallowing or deleting the back channel OPN from the third nodein the source port is important to dual simplex because it prevents thesource node from receiving it and becoming confused. Because of thestructure of the FCAL protocol, it is a protocol violation if a nodereceives an OPN, and that node has already established a previousconnection. However, when a node has opened another node, it isexpecting possible data frames to be sent back to it from thedestination node on the back channel (the part of the loop not beingused for outgoing data from the source to the destination). Thus, if anode other than the destination node has frames to send to the sourcenode, the following things happen. First, the third node sends an OPN.The port coupled to the third node uses the destination address in theOPN to look up the ID of the port coupled to the destination node. Thissearch results in output from the routing table of the ID of the sourceport coupled to the source node in the original forward channeltransmission. This ID is then used to search the scoreboard memory todetermine the status of the source port. The source port sent a messageto the scoreboard memory on the protocol bus updating the status entryin the scoreboard memory for the source port to busy when the sourceport established the front channel connection. In one alternativeembodiment, this busy status will be found when the scoreboard isconsulted, but that will not deter the third port from posting aconnection request message on the protocol bus requesting connection tothe source port. The source port will determine if it has any storedbuffer credit and grant the request if it does by posting a replymessage on the protocol bus naming the backplane channel to use. Thethird port and the source port then both connect to the named backplanechannel.

[0114] There are several alternative embodiments for establishing theback channel simplex connection through the backplane. One involvesupdating all the scoreboards of all ports with information as to whichsource node address is coupled to any port which is indicated in thescoreboard as having a busy status and is thus available for dualsimplex. In this alternative embodiment, the source port posts a messageto the protocol bus to update all scoreboards in every port to indicatethat although it is busy, it is coupled to the source node of the looptenancy and is thus available to receive data in a dual simplex mode. Inthis alternative embodiment, the third port checks its scoreboard, andif it determines that its destination node has the same address as thesource node coupled to the source port, it then sends a connectionrequest message. If it determines from the scoreboard data that thesource port is not dual simplex capable, it does not send a connectionrequest message.

[0115] In the preferred embodiment each switch port which utilizes dualsimplex is only coupled to one node, because the switch ports are notexpensive. In this preferred embodiment, the scoreboards are not updatedwith data indicating the address of the node connected to the sourceport so as save memory space in the scoreboards and traffic on theprotocol bus. In the preferred embodiment, the third node simplytransmits a connect request on the protocol bus naming the source port.The source port then determines if it is dual simplex capable bychecking its own scoreboard for data indicating whether it is dualsimplex capable and determines if its camp list is full. If it dualsimplex capable and its camp list is not full, it grants all connectionrequests. It then picks a backplane channel for the back channelconnection and sends a response message on the protocol bus to the thirdnode. Both the third node and the source node then establish aconnection on the identified channel and the third node sends its OPN.The destination address in the OPN is then compared by the source portto the address of the source node using the latched OPN received fromthe source node.

[0116] If there is no match, the source port generates a CLS and sendsit to the third port. If there is a match, dual simplex mode is allowed,the OPN from the third port is quashed, and the stored RRDYs are sent tothe third port from the source port, one RRDY at a time up to the numberof RRDYs stored. In some alternative embodiments, the RRDYs are notactually stored but are counted and the count is maintained. This startsthe transmission of data frames from the third node. The arrival of adata frame from the third node triggers release of another RRDY from thesource port if another RRDY is available at the source port for release.These data frames are received by the source port and transmitted on theback channel to the source node. Any intervening nodes pass the dataframes through because they do not control the loop. Any subsequentRRDYs output by the source node are intercepted by the source port andtransmitted on the backplane channel to the third port to cause thethird node to transmit a data frame for each RRDY so intercepted. When aCLS is received from the third node, any remaining stored RRDYs arepreserved for use as buffer credit for the next third node that wishesto send data to the source node. The job of managing buffer credit fallsto the state machine in every port in the preferred embodiment.

[0117] Another embodiment for the source port to determine if it is dualsimplex capable is to send post a message on the protocol businstructing any port that has an OPN send the destination address of theOPN to the source port for comparison to the source node address. Ifthere is a match, a reply message is posted directing the port to use aspecified backplane channel and both the third port and the source portconnect to that channel. The transaction then proceeds as above.

[0118] The dual simplex process essentially tricks the link layer of thesoftware of the source node into thinking the incoming frames are fromthe destination node, when they are actually from the third node. Sincethe frames internally contain all the information needed to get the datatherein to the right place in the source node, no harm is done becausethe frames pass through the link layer and the upper layers of softwareread the frames and use them properly. However if an OPN from the thirdnode were to arrive on the back channel, the link layer of conventionalNL nodes becomes confused because it knows the source node just openedsome other node and is not supposed to be receiving any OPNs until thecurrent loop tenancy is terminated.

[0119] The source port also mixes these frames of data from the thirdnode in with RRDY primitives transmitted from the destination node ofthe original loop tenancy on the FCAL net back channel. Each time thesource node receives an RRDY from the destination node, it outputsanother frame of data to its destination node. The mixing in of RRDYsfrom the destination node with the frames of data from the third node onthe FCAL net backchannel. causes the source node to continue outputtingdata frames bound for the destination node.

[0120] Closing a dual simplex connection properly to avoid deadlock andsequence errors is important. The state machine of any port coupled to asource node and facilitating a dual simplex transaction (elsewhereherein sometimes referred to as a dual simplex port or source port) mustinsure that when a CLS is output by the source node that the CLS is notforwarded to the destination node until the third node has finishedtransmissions on the back channel and output its own CLS. Further, thestate machine of any port coupled to a source node and facilitating adual simplex transaction must not wait for a CLS in the case whereaccess to the source node is denied to a third node because of the lackof any RRDYs transmitted by the source node to avoid possible deadlock.Thus, the state machine in a dual simplex port must therefore have thefollowing behaviors as illustrated in the following discussion of fourpossible dual simplex close scenarios.

[0121] 1) A CLS is received from the source port—the CLS is replicatedby the source port and sent to both the third port and the destinationport. The third port transmits the CLS to the third node whichhandshakes with its own CLS which is returned to the source port. Thesource port deletes the CLS from the third node but remembers that ithas arrived and releases the back channel connection. The CLS reachesthe destination port and the destination node. The destination nodehandshakes by emitting its own CLS which is transmitted back to thesource port and to the source node which closes. The source port dropsthe front channel connection.

[0122] 2) A CLS is received from the third node—the source port deletesthe CLS so that it does not get forwarded to the source node and sends aCLS generated in the source port back to the third port, and then dropsthe back channel connection to the third port. The third port forwardsthe CLS to the third node. Any new third port can then initiate a newdual simplex connection.

[0123] 3) Destination node transmits a CLS—the CLS is received at thesource port and held by the source port. A copy of the CLS istransmitted to the third port which transmits it to the third node. Thethird node closes under normal FCAL close protocol and handshakes bytransmitting a CLS to the source port. The source port transmits the CLSto the source node which closes under normal FCAL close protocol rulesand handshakes by transmitting a CLS to the source port. That CLS isforwarded by the source port to the destination port which forwards itto the destination node. The source port then drops both the front andback channel connections simultaneously.

[0124] 4) Destination node transmits a CLS—the CLS is received at thesource port and held by the source port. A copy of the CLS istransmitted to the third port which transmits it to the third node. Thesource node closes under normal FCAL close protocol and handshakes bytransmitting a CLS to the source port. The source port holds the CLSfrom the source node until a CLS is received from the third node. Uponreceipt of a CLS from the third node, a CLS generated in the source portis simultaneously transmitted to the destination port and the sourcenode. The source port then drops both the front and back channelconnections simultaneously.

[0125] Specific Dual Simplex Example

[0126] As a specific example of dual simplex in the context of FIG. 5using a scoreboard which indicates dual simplex capability, refer toFIGS. 16A to 16D which are a flowchart of the processing carried out bythe ports to implement dual simplex communication. In this example,suppose node 138 want to send data to node 101. Suppose also that node144 wants to send data to node 138. Node 138 generates an OPN(101, 138),RRDY, RRDY and transmits these three primitives to port 126 (step 360).Port 126 converts the OPN to a simplex OPN(101, 101) and stores theoriginal OPN including both the source and destination addresses of theOPN and stores the two RRDYs (steps 362 and 364). Port 126 looks updestination address 101 in the routing table and finds it is connectedto port 128 (step 366). The scoreboard table is consulted, and port 128is found to be available (step 368). Port 126 posts a message on theprotocol bus 121 requesting port 128 to initiate arbitration for itslocal loop and pick a backplane channel (step 368). Port 128 winscontrol of its loop and sends a reply message naming the backplanechannel 1 for use (step 370). Ports 126 and 128 send commands to theirportions of the distributed crossbar switches to connect to backplanechannel 1 (step 372). Port 126 forwards the OPN(101, 101) to port 128 assoon as node 128 is available and it passes straight through port 128without delay to node 101 (step 374). The two RRDY primitives of buffercredit transmitted by node 138 and intended for node 101 are strippedoff the transmission to node 101 by port 126 and maintained as a countin the source port for award by regeneration and transmission to anothernode that wants to ship frames to node 138. Node 101 replies with anRRDY (step 376). This RRDY is transmitted back on backplane channel 1 toport 126 where, as described below, it is mixed in with data frames froma third node (node 144) that is transmitting data frames in dual simplexmode to node 138 so as to keep the source node 138 outputting new framesto the destination node 101.

[0127] In the embodiment considered here, port 126 posts a message tothe protocol bus to update all scoreboards that it is busy but isavailable for dual simplex connections. This message can be based uponthe fact either that the source node knows by watching traffic orcomparing source node addresses on its local FCAL net to the AL-PS valuein the original OPN, that it is connected to a source node of the frontchannel and is capable receiving dual simplex transmissions, or itsconfiguration bit indicates dual simplex connections are allowed. Thismessage on the protocol bus would would result in update of allscoreboards (step 380). Node 114 then arbitrates for and wins control ofits FCAL net and generates an OPN(138, 144). Port 124 receives this OPN,stores it (and converts it to half duplex in the preferred embodiment)and uses the destination node address 144 to search the routing table(step 384). This returns the ID of port 126. This ID is used by port 124to search its scoreboard table. Port 124 finds port 126 in its routingtable, finding from its scoreboard that port 126 is busy but dualsimplex capable (step 386) and posting a connection request on protocolbus 121 requesting a connection to source port 126 (step 388). Thiswould result in a connect response reply message from port 126 grantingthe request and naming backplane channel 2 for use (step 390). Bothports 124 and 126 would connect to channel 2 (step 392), port 124 wouldsend its OPN(138, 144) to port 126 (step 394) which would cause port 126to compare the destination address 138 to the source node address (step396), find a match, quash the OPN and forward one RRDY to port 124 viachannel 2 (step 400). The backchannel transaction would proceed untilbuffer credit ran out, or third node had no more data frames to send ora CLS was sent by either node 138 or 144 (step 402).

[0128] In the preferred embodiment, port 124 would simply post aconnection requests which would automatically be granted by port 126with a reply message saying use backplane channel 2. Ports 124 and 126would both connect to backplane channel 2, and port 124 would send itsOPN(138, 144) to port 126. Port 126 would compare the source address itlatched from the original OPN, which is 138, to the destination addressof the OPN received from node 124, which is also 138, and find a match.Port 126 would quash the OPN(138, 144) and transmit one RRDY back toport 124 which would reach node 144 and result in transmission of oneframe of data. That frame of data would pass through backplane channel2, port 126 and the back channel path and reach node 138. Port 126 wouldthen send the remaining RRDY to port 124 which would result in anotherframe of data being transmitted. Any further RRDYs transmitted by node138 would be latched or counted and sent to port 124 in the normalcourse of flow control, or if node 144 sent a CLS, the RRDYs stored byport 126 would be saved for grant to another third node until such timeas the original loop tenancy was terminated with a CLS from either thesource node or destination node.

[0129] In alternative embodiments, the port 126 can determine whether itis capable of dual simplex by a sending a message to port 124 asking forthe destination address from the OPN (138, 144) and then compare thisdestination address to the latched source address from the originalOPN(101, 138). If there is a match, sending a reply message saying, “Usebackplane channel X and send me your OPN” whereupon the transactionproceeds as defined above. If there is no match, generating a CLS andsending it to the third port or sending a message that causes the thirdport to generate a CLS and send it to the third node.

[0130] Returning to the main example, port 126 mixes RRDYs received fromnode 101 on the front channel connection with data frames received fromnode 144 and sends the combined data stream to the source node over theback channel of the FCAL net coupled to the source port (step 404). Thiskeeps source node 138 outputting data frames to destination node 101.

[0131] There are three scenarios for closing the dual simplex datapaths. An example of each will be discussed in the next threeparagraphs.

[0132] Suppose source node 138 issues a CLS (step 406). In that case,source port 126 duplicates the CLS and forwards it to destination port128 and third port 124 simultaneously (step 408). The third node 144replies with its own CLS which is forwarded to port 126 (step 410). Port126 discards the CLS received from third node 144 but notes that it hasarrived and both ports 124 and 126 relinquish their back channelconnections through the backplane (step 410). When the CLS reachesdestination node 101, it replies with its own CLS back to source node138, and ports 126 and 128 relinquish their front channel connections(step 412).

[0133] Now suppose third node 144 issues a CLS (step 414). In this case,the CLS is transmitted to source port 126 which deletes the CLS (it isnot sent to the source node 138—step 416), and ports 124 and 126 eachrelinquish their back channel connection through the backplane (step418). Any new third node can initiate a new dual simplex connection.

[0134] Finally, suppose destination node 101 issues a CLS (step 420).Source port 126 receives the CLS and holds it temporarily and does notsend it to source node 138 (step 422). A copy of the CLS is sent tothird port 124 and the third node (step 424). The third node replieswith its own CLS which is received by the source port and sent to thesource node (step 424). The source node replies with a CLS which isforwarded to the destination node (step. 426). The source port and thedestination and third ports then drop the front channel and back channelconnections (step 428).

[0135] Dual simplex mode allows many different third nodes to deliverframes to the source node during its “conversation” (loop tenancy) withthe original destination node. Dual simplex almost doubles throughput ofthe switch.

[0136] The Process of the Preferred Embodiment

[0137] An example of normal, non dual simplex processing by the switchillustrated in FIG. 5 (or any of the alternative embodiments of non dualsimplex switch operation) is illustrated by the flowchart of FIGS. 6Athrough 6C. This example is only a high level illustration of one paththrough the states of the loop port state machine of a single switchport such as loop port state machine (LPSM) 218 in FIG. 7 and isillustrative of typical processing for loop-local or chip-local orremote transactions. The LPSM in each switch port is a complex statemachine which has many states and many transistions between states withthe transitions between states depending upon the logical states ofvarious input signals. Each loop port state machine is comprised of aplurality of individual component state machines which interact witheach other. Some state machines generate output signals which are inputsto other state machines. This means that a large number of possiblescenarios for the states of the state machine switch ports exist. Toexplain all these states and all the conditions for transitiontherebetween, would obscure the main ideas of the invention. To exactlyillustrate all possible states and transitions, there is includedherewith as FIGS. 17 through 28 individual state diagrams for all thestate machines in a single LPSM. Each circle represents one state. Eachline from one state to another represents a transition from one state toanother. The labels in boxes on each line represent the Booleancondition that must exist between the signals identified in the box forthat transition to be made. Boolean logical operators are identified by& for an AND operation, a tilde preceding a signal name means NOT and avertical line between two signals means OR between those two signals.Each state and each signal is defined in the verilog code appendedhereto as Appendix A. The state diagrams coupled with the verilog codetogether comprise a complete and exact description of all states andtransitions between states for every possible processing scenarioperformed by the preferred embodiment of the switch. A description ofthe verilog hardware description language and how to use it to definethe functionality of an integrated circuit is given in the treatise,Palnitkar, “Verilog® HDL: A Guide to Digital Design and Synthesis” ISBN0-13-451675-3 (Prentice Hall 1996) which is hereby incorporated byreference.

[0138] Local Destination Process

[0139] As a first example, suppose node 138 wishes to send data to node140. Node 138 arbitrates for control of the FCAL net 142 comprised ofdata paths 142 A, B and C and nodes 138 and 140 (step 150). When controlis won, source node 138 sends an OPN primitive to port 126 with thedestination address of destination node 140 therein (step 152).Optionally, the source node follows the OPN with one or more RRDYprimitives, each representing one frame buffer of credit which thesource node has reserved for receiving frames of data from thedestination node (step 152). Port 126 latches the OPN and any RRDYprimitive following the OPN (step 154), and uses the destination addressof the OPN as a search key to search a routing table stored in port 126(step 156). In the preferred embodiment, a separate routing table ismaintained in each port circuit. In alternative embodiments such asrepresented by FIG. 4, the routing table may be a single table 127coupled to all the ports by the protocol bus 121 with each port checkingthe routing table contents by messages on the protocol bus. Thisembodiment is represented by FIG. 4. In the embodiment of FIG. 4, thecontents of the routing table are updated by an active discovery processalthough this active discovery process may also be carried out in thepreferred embodiment of FIG. 5.

[0140] The contents of the routing table are destination addresses andthe identification codes (hereafter IDs) of the ports or half bridges towhich those destination nodes are coupled. Each half bridge containssuch a routing table. The contents of the routing table are learned byeach half bridge by watching the traffic on its ports in the mannerdescribed in the parent case or by an active discovery process whichwill be described below.

[0141] In this particular case, the search indicates that thedestination node is local to FCAL net 142, and that determination isrepresented by test 158 in FIG. 6A. Port 126 responds by setting aninternal switch to connect an internal bypass data path to connect datapath 142C to data path 142A (step 160), and sends the OPN to destinationnode 140 along with any RRDYs received from the source node (step 162).Node 140 responds either with one or more RRDY primitives if it hasbuffer capacity to receive one or more frames from the source node or aCLS primitive or one or more frames of data to be transmitted to thesource node in accordance with the number of RRDY primitives receivedfrom the source node (step 164). In general, step 164 represents theprocess of the switch port acting as a normal non participating FCALnode for the duration of the transaction by forwarding all data andprimitives received from the local loop back onto the local loop. Thus,if a primitive or data frame is output by the destination node fortransmission on data path 142B to source node 138 and node 138 respondswith an RRDY or data frame on path 142C, switch port 126 forwards theRRDY or data frame to the destination node via path 142A. If thedestination node outputs an RRDY, source node 138 responds bytransmitting an entire frame to node 140 via data paths 142C and 142Aand the internal bypass data path (not shown) within half bridge switchport 126 (or a number of frames equal to the number of RRDY primitivesreceived from the destination node may be transmitted—step 164). Theframe is stored in node 140, and when it has been processed and thebuffer is ready to receive another frame, another RRDY is output. If thedestination node responded with a frame of data in response to an RRDYreceived from the source node, the source node stores it in its buffer.If the destination node responded with a CLS primitive, the source noderelinquishes control of its FCAL net, and the transaction is endedaccording to normal FCAL close protocol. The process of exchanging RRDYsand frames of data continues until either node 140 outputs a CLSindicating it will send no more data to the source node or source node138 outputs a CLS indicating it will send no more data to thedestination node. The FCAL net 140 is then relinquished and other nodesincluding port 126 can arbitrate for control thereof.

[0142] Test 166 is symbolic of one event that can occur after theloop-local transaction is completed. The loop-local transaction has tobe completed since no other node can win control of the local loop inarbitration so as to be able to send an OPN until the loop localtransaction is over. Test 166 determines if an OPN has been receivedfrom the local loop, and, if so, vectors processing to step 154 to latchthe OPN, as symbolized by step 168.

[0143] Test 170 symbolizes a test for an event which can occurregardless of whether the loop-local transaction is over ornot—receiving a connect request message on the protocol bus at theswitch port involved in the loop-local transaction. If this happens, theswitch port becomes the destination port for a remote transaction. Insuch a case, the switch port behaves as indicated in step 175 and thefollowing steps to arbitrate for control of the local loop, assymbolized by block 172. When control is won (after the loop-localtransaction is over), the switch port sends back a connect responsemessage naming the backplane channel to use. The connection is thenestablished by the source port and the destination port and thetransaction is completed as indicated in the steps following step 175.

[0144] If tests 166 and 170 do not detect either a new OPN from thelocal loop or a connect request on the protocol bus, the loop port statemachine loops back to test 166 and stays in the state represented bytests 166 and 170 until one of the events detailed there happens.

[0145] Remote Destination Process

[0146] Now suppose node 140 has data to send to node 144 coupled to port124. Node 140 arbitrates for FCAL net 142, and when control is won,outputs an OPN with the destination address of node 144 therein (steps150 and 152, FIG. 6A). The source node may also output one or more RRDYprimitives. The OPN and any RRDY primitives are latched in the localnode (step 154), and the local node uses the destination address in theOPN to search the routing table (step 156). In this case, search of therouting table indicates that node 144 is coupled to a remote port 124,and an ID for the remote port is returned by the search (step 158). Ifthe search indicated the destinaton node was on an FCAL net coupled to aswitch port on the same chip as the switch port coupled to the sourcenode, the result would be the same in that processing would be vectoredin either event to step 168 169. Thus, path 167 is taken for either achip-local or a remote transaction.

[0147] Step 168 represents a determination as to the status of theremote port coupled to the destination node. The combination of steps168 and 176 represent a determination of the status of the remote portand branching to an appropriate routine depending upon the results. Ifthe remote port status is “available” (path 170), source switch port 126then sends a connect request message formatted as in FIG. 14A via aprotocol bus 121 to the destination or chip-local switch port, assymbolized by step 172. In FIG. 14A, the requestor ID is the ID of thesource switch port and the responder ID is the ID of the destinationswitch port. Although this is the protocol of the preferred embodiment,in alternative embodiments, any protocol for determination of the statusof the remote port and/or any other mechanism to establish a connectionthrough the crossbar switch or other connection mechanism will sufficeto practice the invention. In the preferred embodiment, the statusdetermination is accomplished by checking the status entry for thedestination port in a copy of a scoreboard table stored by the switchport and kept up to date with the copies of the scoreboard table in allother switch ports by monitoring messages on the protocol bus. Inalternative embodiments, the local port can send a message to the remoteport via the protocol bus asking it for its status or can send a messageto a central scoreboard to get the status of the destination port.

[0148] If the remote port was busy, path 174 is taken from the statusdetermination test 176 back to step 168 to read the status again andprocessing stays in this loop in the local port until the status changesto available or no privilege. In some embodiments, a timeout functioncan be included to exit the busy-wait loop path 174 and send a CLS tothe source node after a timeout event to keep head end blocking frombarring all communications will all nodes coupled to the local port incase the remote node is broken and that would be equivalent to theclaimed invention.

[0149] If test 176 determines there is no privilege for the source nodeto talk to the destination node, path 178 is taken to block 180 wherethe local port sends a CLS to the source node. Processing then returnsto Start.

[0150] Assuming the remote port was available and the connection requestwas received, the remote port then starts arbitrating for control of itslocal FCAL net (step 175, FIG. 6B). When control is won, the remote portsends back a response message informing the first port coupled to thesource node to forward the OPN and any RRDY primitives it has latched,and telling the first port which backplane channel to use (step 177).

[0151] As symbolized by block 179, the response message from the remoteport causes both the first port and the remote port to generate commandsto the distributed crossbar switch circuitry of each port to setswitches to couple the two ports to the selected backplane channel toset up an FCAL loop connection between the first port and remote portthrough the crossbar switch. In the example at bar, this data pathcouples port 126 and port 124 through the backplane data path 130 on thechannel named in the reply message (step 179).

[0152] In some alternative embodiments, instead of sending a connectionrequest to the remote port, the first port will simply establish a datapath through the backplane and send the latched OPN and any followingRRDYs to the remote port 124. In response, port 124 will then latch theOPN and any following RRDY primitives, arbitrate for control of the FCALnet coupled to node 144, and when control is obtained, transmit the OPNand any following RRDYs to destination node 144. If the local FCAL netis busy, the remote port will hold the OPN and any following RRDYs untilthe FCAL net is available, and repetitively arbitrate for control andthen, when control is achieved, forward the OPN and RRDYs to thedestination node.

[0153] In this alternative embodiment, the destination node will respondto the OPN (and any following RRDYs) with either an RRDY if it hasenough buffer space to receive a data frame or a CLS if it does not orwith one or more data frames to be transmitted to the source nodecorresponding to the number of RRDY primitives received from the sourcenode if any RRDY primitives were received. The transaction then proceedsby transmitting one frame for every RRDY received from the other nodeuntil one or the other of the source or destination nodes transmits aCLS primitive. The frame or frames stream through ports 126 and 124 andbackplane data path 130 without ever being stored in any buffer memoryand without ever being segmented into packets that require segmentationand reassembly.

[0154] In the preferred embodiment, port 126 instead of sending the OPNand any RRDYs to port 124 to signal it to establish a connection,instead sends a connection request message via the protocol bus 121. Theconnection request message tells the remote port 124 to arbitrate forits local loop and to notify port 126 when control is won and expectfurther traffic. When the response message is received that control hasbeen won by port 124 of its local loop and naming a channel to use onthe backplane for data transfer, the OPN and any RRDY primitives whichwere received by the first port from the source node are sent to port124 which then forwards these primitives to node 144 (step 181). As wasthe case for the alternative embodiment discussed in the paragraphabove, the destination node will respond with either an RRDY if it hasenough buffer space to receive a data frame or a CLS if it does not orwith one or more data frames to be transmitted to the source nodecorresponding to the number of RRDY primitives received from the sourcenode if any RRDY primitives were received. The transaction then proceedsby transmitting one frame for every RRDY received from the other nodeuntil one or the other of the source or destination nodes transmits aCLS primitive to complete the transaction.

[0155] The connection is closed with a handshaking type protocol thatuses a switch control character to eliminate race conditions. Theconnection through the backplane can be closed by either the source nodeor the destination node sending a CLS. Test 182 represents branching todifferent protocols for closing depending upon whether the source nodeor destination node initiated the CLS. Processing branches to step 183is the source node initiated the CLS. In step 183, the source nodeoutputs a CLS and that CLS is received by the source port and forwardedto the destination port over the backplane connection. Optionally, thesource port also sends an ARB(F7) or other innocuous fill word to thedestination port. The purpose of this fill word is to send affirmativecharacters to the destination port to prevent any spurious charactersfrom being transmitted to the destination port.

[0156] Step 185 represents the process of receiving the CLS at thedestination port and forwarding it onto the local loop so that thedestination node receives it. The destination node responds with its ownCLS.

[0157] Step 187 represents the process of receiving the destinationnode's CLS at the destination port and forwarding it acrossthe-backplane connection to the source port followed by a switch controlcharacter (typically 20 consecutive zeroes).

[0158] Step 189 represents the process of receiving the destinationnode's CLS at the source port and forwarding it onto the local loop sothat it will be received by the source node. The source port also loopsthe switch control character back through the backplane connection tothe destination port.

[0159] Step 191 represents the process carried out in the destinationport of receiving the switch control character transmitted by the sourceport and realizing the backplane channel in use is now clear. Thedestination port then drops the backplane connection and transmits arelease frame on the protocol bus. All other ports see that releaseframe and update their scoreboards to indicate availability for thatbackplane channel. The switch ports involved in this transaction are nowready for a new transaction so processing returns to start state 148.

[0160] In the case where the destination node initiates the CLS, test182 vectors processing to the closing sequence which starts with step195. Step 195 represents the process of the destination node initiatinga CLS. The CLS is received by the destination port and forwarded acrossthe backplane connection to the source port followed by a switch controlcharacter (currently 20 consecutive zero bits).

[0161] Step 197 represents the process that happens at the source portwhen the CLS and switch control character is received by the sourceport. The source port forwards the CLS onto the local loop and holds theswitch control character. The CLS reaches the source node, and itresponds with its own CLS. This CLS is received by the source port andforwarded to the destination port across the backplane connection. Afterforwarding the CLS onto the backplane connection, the switch controlcharacter is sent across the backplane connection. The source port thencloses its backplane connection.

[0162] Step 199 represents the process which occurs at the destinationport when the CLS and switch control character are received there. TheCLS is forwarded onto the local loop by the destination port. There itis received by the destination node which causes the destination node toclose. The destination port then notes that it has received back theswitch control character it originally sent to the source port andcloses its backplane connection. The transaction is then over and thetwo switch ports are ready for a new transaction. Processing thusreturns to a start state represented by block 148 as symbolized by step193

[0163] Note in all of the above embodiments, the frame or frames streamthrough ports 126 and 124 and backplane data path 130 without ever beingstored in any buffer memory and without ever being segmented intopackets that require segmentation and reassembly.

[0164] The lookup table in each port circuit is slightly different thanthe lookup table in the bridge described in the parent described in theparent application. There, only two half bridges were coupled together,so the routing table could output only a 1 if the destination was localor a 0 if it was not, or vice versa. With the many half bridges of theswitch, the routing table must store more bits for each destinationaddress, those bits (6 ID bits plus one bit for validity of the port)define the unique ID of the particular port to which each node having aparticular destination address is coupled.

[0165] The Scoreboard Table

[0166] In the preferred embodiment, a scoreboard table is used to storeport status and priviledge information. Whenever the local FCAL net of aport is busy, the port connected to that FCAL net sends a message toupdate its status in the scoreboard to “busy”. When the FCAL net becomesavailable again, the port connected to that net sends another message tothe scoreboard to update its status to “available”. The scoreboard canalso store privilege information to implement security rules to preventcertain ports from communicating with other ports. The status states mapto the following actions by the switching circuit: “busy camp” equalswait and do not forward OPN so no RRDYs get back to source node and nodata is sent; “available” equals connect and send OPN to destination;and “busy no camp” equals generate a CLS and send it back to the sourcenode, and “no privilege” equals generate an OPN and send back—this willtell the source node that the destinat node is not available.

[0167] Summary of the Preferred Routing Algorithm Using a Scoreboard andProtocol Bus

[0168] Thus, in the local and remote destination methods describedabove, the preferred embodiment includes the following steps forconsulting the scoreboard (this method is reflected in FIGS. 6A and 6B).

[0169] After the OPN and any following RRDYs from the source node arelatched, the port that latched the OPN determines the port ID of theport coupled to the destination node by consulting the routing table asdescribed above. After determining the ID of the remote destinationport, that port ID is used as a search key to consult the scoreboardtable to determine if the port is available and if there is theprivilege to talk to it. If the remote ports status is “busy camp” andcamping is allowed, send a connect request message and wait for theremote port to finish its current conversation, grant the connectrequest and send back a connect response message naming the backplanechannel to use. If the remote port's status is “no privilege”, generatean OPN and send it back to the source node. If the remote port's statusis “available”, send a connect request message to the remote port overthe protocol bus. If the status if “busy no camp”, return a CLS to thesource node. This causes the remote port to arbitrate for and wincontrol of its local loop. When control is won, the remote port thensends a reply message to update its scoreboard status to busy and namingthe backplane channel to use. The scoreboard circuitry sees the replymessage and updates the status of the port whose ID is in the replymessage to busy. The reply message causes the first port to generatesignals to the crossbar switch to open a connection between the twoports. The first port then sends the latched OPN and any latched RRDYsto the second remote port coupled to the destination node, and theprocess proceeds as described above until the transaction is completed.When the transaction is completed, both ports relinquish control oftheir FCAL nets and the destination port posts a message on the protocolbus that both the source and destination ports are availble. Thescoreboard circuitry sees this messages and automatically updates thestatus of each of these ports to “available”.

[0170] When search of the routing table with the destination address ofthe OPN received at the first port from the source node indicates thatthe destination node is local, the scoreboard is not consulted and thebypass data path is activated. Then a message is sent by the first portto the scoreboard updating its status to “busy” to let all other portsknow that the FCAL net coupled to the first port is temporarilyunavailable for any other loop tenancy. Of course camping is allowed onlocal connection in the case of a “busy camp” status.

[0171] The status data in the scoreboard table will indicate a port isbusy when the FCAL net coupled to the port is tied up in a loop tenancy.

[0172] Referring to FIG. 7, there is shown a block diagram of a typicalswitch chip such as those represented by blocks 124, 126 and 128 in FIG.5. Three FCAL net ports, each having a transmit and receive terminal,are shown at 200, 202 and 204. Each port has an FCAL net interfacecomprised of a 1 Gbit/sec serializer/deserializer shown at 206, 208 and210. The function of these SERDES interfaces is to transmit and receiveserial data, converting the serial data stream to and from 10-bitcharacters used internally in the switch chip.

[0173] The received 10-bit data data is re-timed in an elastic buffer,shown at 212, 214 and 216, producing a 10-bit data stream whose timingis identical to the transmitted data.

[0174] A loop port state machine (LPSM) for each port, shown at 218,220, 222, functions to process received FCAL data and generate commandsto establish the proper connection to either the local FCAL net or to adesignated channel through the backplane so as to facilitate coupling ofthe received data on whatever data path the state machine establishes.The state machine also functions to receive data from the crossbarswitch coming from a remote port and couple it onto the FCAL net localloop of that port. The state machine also functions to do the routingtable searches and scoreboard table searches to determine when and FCALtransaction from the local loop must connect to the switch fabric and topost transaction messages on the protocol bus such as connectionrequests, reply messages, scoreboard updates etc. The switch fabricrefers to the backplane data paths and switching circuitry that allowsconnections between different ports. The state machine also functions toarbitrate for control of the local FCAL net when a connection request isreceived from the switch fabric, and, when control is achieved, pick abackplane channel and generate the reply message. Another function ofthe state machines in every port in dual simplex mode is to store thebuffer credit RRDYs output by the source node and forward these RRDYs tothe third port, one RRDY at a time up to the buffer credit limit, whenan OPN for a dual simplex transmission is received from a third node.Basically, all the logic and intelligence of each hub to carry out thefunctions described herein for the preferred or alternative embodimentsresides in the states of the state machine. The particular design of thestate machine is not critical, and any state machine (or programmablemachine if it is fast enough such as a microprocessor) which can performthe functions defined herein will suffice to practice the invention. Thenovelty is not believed to be in the particular design of the statemachine but in the functions it performs in each port.

[0175] Since each switching circuit such as 124 in FIG. 5 contains threeports (only one port is shown in FIG. 5), each switching circuitcontains a port multiplexer, shown at 224 in FIG. 7. The portmultiplexer interfaces the three state machines or LPSMs to thebackplane data channels 226. The state machines control the multiplexer224 to keep local destination traffic for each port local but to coupletraffic between the port and a remote port to a channel on thebackplane. The port multiplexer and the state machine jointly determinethe nature of each new conversation (local or remote) by searching arouting table stored in memory 228. Preferably, memory 228 is a lookuptable.

[0176] The backplane 226 comprises the datapath and control logicrequired to receive and transmit the backplane data between switchingcircuits as well as multiplexing circuitry to shunt backplane data toand from the state machines via the port multiplexer 224. Backplaneconnections and data channel availability are tracked by a backplaneprotocol circuit which manages scoreboards in each port which have theirdata synchronized via broadcasts over the protocol bus 121.

[0177] The state of each switch chip is monitored by a managementcircuit that interfaces to the system through 2-wire bus 232. JTAGcircuit 234 is used for testing the switch chip.

[0178] The serializer/deserializer function (hereafter SERDES) of eachport at 206, 208 and 210 is provided by the GigaBlaze SerialLink™megafunction circuitry which is commercially available from LSI logic orany equivalent SERDES circuitry, the details of which are herebyincorporated by reference. The SERDES circuits accepts differential,8b/10b encoded serial data at the rate of 1 Gbit/sec, demultiplexes itinto aligned 10-bit characters and recovers a receive clock from thedata. At the same time, the SERDES multiplexes outgoing 10-bitcharacters into a 1 Gbit/sec differential transmit data stream using aself-contained 10× clock multiplier. Other currently available SERDEScircuits may be used also either at macros in layout of the switch portchips or as external parts.

[0179] The elastic buffers 212, 214 and 216 absorb differences betweenreceive and transmit data rates which can vary up to 100 ppm from thenominal rate of 1.0625 Gbits/sec. The elastic buffers re-time theincoming data stream so that its timing is compatible with the transmitdata. This is achieved by storing received data in a FIFO as previouslystored data is removed at a rate defined by the transmit clock. If thereceive and transmit rates are not equivalent, the receive and transmitpointers will eventually converge. To prevent this, logic senses animpending pointer collision, and repeats or deletes data when the FC-ALfill word data is present in the FIFO. A fill word is a Fibre Channelordered set defined to be insertable or deletable as elasticityrequirements dictate. The FIFO also contains circuitry to perform asmoothing function to repair interframe gaps that have been made toosmall by the FIFO by deletions from a small interframe gap so as to makeit so small that it creates problems downstream. The repair ofinterframe gaps is done by inserting fill words into small gaps andremoving fill words from large gaps. Specifications for such a FIFO areprovided in the FCAL standards documents which are incorporated byreference herein.

[0180] The state machine in each port is similar to the FC-ALspecification state machine but different in that it is passive andnever transmits or receives frames itself. The state machines 218, 220and 222 relay OPNs and frames to remote destinations as well asperforming the other functions defined above. When a state machinereceives data from the backplane, it arbitrates for control of its localFCAL net with high priority according to arbitration rules defined inthe FCAL standards incorporated by reference herein. Once control isachieved, the switch port then relays the OPN and frames onto the localFCAL net.

[0181] The state machines also implement the dual simplex processdescribed above. Dual simplex mode can be turned on or off by managementcommands received on bus 232. When this mode is on, the state machinesconvert full duplex OPNs to remote devices to half duplex OPNs to thesame destination address. This leaves the source port available toreceive an incoming connection from a third port.

[0182] The state machines do not need to participate in loopinitialization, but rather they optionally allow initialization to occuras it would in a prior art FCAL net configuration and then change thestate of the switch to allow the state machine to active engage sourceand destination nodes and control data flow therebetween. Beforeswitching from hub mode (the switch can be instructed to act as a hubinstead of a switch), the state machines also function to learn themapping between each destination node address and the port ID of theport to which that destination node is connected. This learning can beperformed by either a discovery process or OPN trapping, and will bediscussed below. The mappings learned in either process are written bythe state machine to the routing table in memory 228.

[0183] The port multiplexer 224 selectively connects each port to eachother within the switching circuit or to the backplane 226. The portmultiplexer has three multiplexer blocks each of which is controlled bya portmux state machine. Local traffic connections are implemented byconnecting a transmit data path and a receive data path between the twostate machines in the port multiplexer 224. Remote connections areestablished by connecting the transmit data path and receive data pathbetween the state machine and the backplane. Each channel through theport multiplexer actually implements two transmit and receive datapaths. In full duplex operation, only one TX-RX pair is used to carryhalf or full duplex conversation. When a port is configured fordual-simplex operation, the outgoing conversation uses one TX-RX pair totransmit data to and receive flow control primitives from thedestination node while the other pair is used to receive data from andtransmit flow control to any third node in the event a third nodeconnects to the source port.

[0184] The port multiplexer 224 also contains a camplist FIFO for eachof the three ports in the switch chip. These FIFOs are used to recordthe IDs of remote ports that have made a priority request to the switchport but which have not been serviced yet so as to implement thefairness token highest priority to these remote ports which have not yetbeen serviced and prevent any starvation of a port.

[0185] The backplane 226 is a group of parallel point-to-point datapaths which physically comprises 14 2-bit wide, 531.25 Mbit/sec datapaths and a 265.625 MHz strobe signal imlemented using differential LVDSto drive the receiving switch chip. The switch chips are connected suchthat the backplanes form a ring of point-to-point connections. Thestrobe is generated by an integrated PLL 227 and is timed with outgoingdata such that it can be used to latch the data at the destinationswitch chip which is the next chip in the ring. Each data channel on thebackplane may operate in the pass-through mode, or it may be configuredto align incoming data into 20-bit words and steer data to and from aport multiplexer channel, which typically happens when a remoteconnection is being serviced by a port. Each port can be configured tostatically listen and transmit to certain channels to support broadcast.

[0186] The backplane logic tracks the availability of each backplanechannel, and can be configured to associate a data channel with areceiving port and only release it when no other channels are available.This is useful where backplane channels are underutilized to reducelatency.

[0187] The protocol bus is an 18-bit wide data bus used for broadcastingconnection requests and response messages to each switch chip in theswitch. The bus is arbitrated by external logic that is asynchronouswith respect to the main switch chip logic and thus can operate at anyspeed up to the limit determined by the system design. Each switch chipmonitors the protocol bus and updates the status information in its copyof the scoreboard such that every port knows the busy/available statusof every other port. This allows denial of full duplex connections atthe source if the destination port is busy. Dual simplex is an exceptionto this rule. The scoreboard also prevents loops of camped devices bydenying any camping connections to a port that is already camped onanother port.

[0188] The routing table in LUT memory 228 stores 6-bit mapping databetween each destination address and its port ID. Each port in a switchchip must have received a unique ID assignment from external managementsoftware before the system can operate as a switch. The LUT memory hasthree 10-bit read address ports, one dedicated to each port on a switchchip. The LUT has one 10-bit write address port for writing associationsfrom the state machines developed in the learning process and also has7-bit read and write address ports used by management software whichdoes not form part of the invention. Internal logic in the LUT convertsthe 10-bit AL-PA destination addresses to 7-bit addresses for the 128×8SRAM. The output from the LUT is a 6-bit port ID that maps to the AL-PAinput at the address port of the lookup table, a valid entry bit and abit that indicates whether the destination node is local to the switchchip. When a port receives an OPN from its local loop, it uses LUT 228to determined if the conversation will be loop-local (destination onlocal loop), chip-local (destination on the local loop of another porton the same chip) or remote.

[0189] The LUT is initially loaded by a learning process called OPNtrapping in which each port observes OPNs on its outbound local loopport and writes a destination address to LUT if a response to theoutbound OPN on the local loop is detected on the local loop inbounddata path. The port then posts a message on the protocol bus that itslocal loop has the destination address of the outbound OPN and givingits switch port ID. All the other ports receive this message and writethe mapping into their routing tables.

[0190] The contents of the LUT are cleared by a state machine andcounter triggered by a managment command or the rising edge of a chipreset signal. Clearing the LUT takes 128 clock cycles.

[0191] Operational Overview

[0192] The switch chips of FIG. 7 can be operated in three fundamentalconfigurations: hub, switch and locked. Initial operation is in hub modewhere each port is coupled to its neighbor ports to form one big FibreChannel Arbitrated Loop (FC-AL). In locked mode, each port is standingby in preparation for a mode switch or broadcast. In switch mode, eachport forwards data back onto its local loop until an OPN to a remotenode is detected, and then a connection to the port coupled to theremote node is initiated through the backplane.

[0193]FIG. 8 illustrates hub mode. In hub mode, the state machines ofall ports cooperate to forward loop data from the receiver of one portto the transmitter of the adjacent port through the port multiplexer224. Because each switch chip can specify to the management softwarethrough a management-programmable register which hub mode backplanechannel it uses, it is possible to partition the switch into severaldistinct loops instead of one big one. The hub channel is selectable ona per port basis.

[0194] Switch mode the state machine of each port monitors traffic onits local loop and forwards data from its receiver to its transmitterwhen an OPN designating a remote node is received on the local loop.When this happens, the state machine breaks the local loop andsubstitutes fill words on the local loop transmitter so that nodes onthe local loop do not know the loop has been broken. The OPN is heldwhile a connection attempt to the remote port is made. If theconnection, can be made, the remote port conducts unfair arbitration towin control of the remote loop, breaks that loop if it wins control andtransmit and receive connections between the two loops are initiated toform a loop across the backplane between the source and destinationnodes. FIG. 9 shows a typical switch connection between two ports ondifferent chips as data path 250, a simultaneous loop-local conversationrepresented by data path 252 and a simultaneous chip-local conversationas data path 254.

[0195] At any time, a number of these connections may exist with themaximum number of chip to chip data paths limited by the number ofbackplane channels.

[0196] Transactions that are purely loop-local proceed withoutintervention of the state machine in the preferred embodiment with theexception that the state machine momentarily holds the OPN receivedwhile it checks the destination address in the LUT 228. If thedestination is on the local loop, then the OPN is forwarded onto thelocal loop and the state machine transitions into the busy monitoringstate.

[0197] The state machine also detects a local tenancy by monitoring theRRDY primitive, and the tenancy is terminated when a CLS is detected. Atthe time a CLS is detected, the state machine returns to either amonitoring state or to a remote arbitration state if a pending remoterequest exists in the camp on FIFO (not shown in FIG. 7).

[0198] Chip-local transactions are handled by the port multiplexer 224by arbitrating for local resources and physically multiplexing the datafor all chip-local conversations. The state machine signals the portmultiplexer and forwards the destination address after receiving an OPNfrom the local loop naming a destination on the same chip but adifferent FCAL net. A hit on one of the other two ports in the chipinitiates the transmit request sequence.

[0199] The transmit requests to chip-local ports must be broadcast onthe protocol bus to resolve port contentions and deadlock issues. Localrequests proceed normally without posting to the protocol bus as nodeadlock is possible for local only transactions.

[0200] When a remote destination address is detected in an OPN from alocal loop (done by the port multiplexer in some embodiments), the portmultiplexer forwards the request to the protocol bus logic 121 whicharbitrates for the bus and assembles a request frame. The destination orremote port receives the frame and either begins unfair arbitration forthe remote loop or queues the request in a FIFO-ordered list of campers.Eventually the request is serviced, and arb on the remote loop is won.At this time, the remote port issues a response frame on the protocolbus containing both grant and backplane channel ID on which to converse.The requesting port releases the OPN once the grant is received andawaits the first valid K28.5 primitive from the chose data channel. Theremote port, upon receiving the OPN primitive, releases its localtraffic onto the data channel and enters the connected state. The remotetenancy is considered terminated when the state machine has detected aCLS from both the source node and the destination node.

[0201] Each state machine implements a port bypass circuit and aparallel loopback mode, and each SERDES implements a serial loopbackmode. The port bypass and parallel loopback modes are also used tobypass a faulty local loop while the switch is operating.

[0202] Port Bypass, Parallel Loopback and Serial Loopback

[0203] Each switch chip has three multiplexers that may be used tocontrol the flow of data without regard to the FCAL protocol asillustrated in FIG. 10. These three multiplexers control port bypassmode, parallel loopback mode and serial loopback mode.

[0204] The port bypass circuit 260, when enabled, connects line 262 fromthe port multiplexer 224 in FIG. 7 back to the port multiplexer via line264 and port bypass switch. This has the effect of forwarding data fromthe port multiplexer 224 back to its source without travelling acrossthe local loop coupled to transmit interface circuit 266 and receiveinterface circuit 268 coupled to the inbound and outbound data paths ofthe local FCAL net. SerDes (serializer/deserializer) circuit 270 doesthe serialization and deserialization function described above. Dataincoming from the local FCAL net is never sent to the port multiplexer224 when port bypass mode is engaged. The port bypass circuit 260 may beengaged by the managment software, and is automatically activated undercertain error conditions. It may only be disengaged by management. Innormal operation, 10-bit words generated by Giga Blaze circuit 271 onoutput 270 are coupled through port bypass switch 260 to the portmultiplexer via line 264.

[0205] Parallel loopback mode is manually engaged and disengaged by themanagement software, and is qualified by the port bypass enable. Thatis, parallel loopback mode is always disabled when port bypass isdisabled. When a switch chip is in parallel loopback mode, incominglocal loop data received by Giga Blaze circuit 271 at input 278 from thereceive interface circuit and output on line 270 is sent immediately tothe transmit interface circuit 266 via line 272, parallel loopbackswitch 274, Giga Blaze circuit 271 and output 276. This happens underall circumstances when parallel loopback mode is engaged. Data from theport multiplexer 224 is never sent to the local loop while parallelloopback mode is engaged. If port bypass is enabled an parallel loopbackis disabled, the loop is broken at the parallel loopback switch andARB(F7) primitives are transmitted onto the local loop.

[0206] Serial loopback mode is primarily used for testing the data pathsin the switch chip.

[0207] Priority and Camping

[0208] All chip-local and remote connections fall into one of threepriority classes: none, low and high. All connection requests from aswitch port default to a static priority level (which is typically nopriority) which can be set my management software. Priority for requestscan be escalated to low or high by a number of mechanisms.

[0209] Priority escalates from none to low after a managementprogrammable number of consecutive denials. If management software sets0 as the number of denials, this escalation is disabled. After thetrigger level is exceeded, the priority deny count is reset and a lowpriority request bit is set for exactly one arbitration interval. Thisarbitration interval begins after the low priority trigger value hasbeen exceeded and an IDLE primitive has been detected on the inboundport of the state machine. All subsequent remote requests will use lowpriority until another IDLE is detected, ending the arbitrationinterval.

[0210] The management software can also directly escalate priority on aport by enabling a rotating-token scheme in which each port requests athigh priority for a period of time while other ports deny requests to aport.

[0211] This mechanism guarantees each port a chance to service its localclients, at the expense of servicing any new remote requests. The tokenposition is computed independently by each switch chip through use of a6 bit token port ID location, a 6 bit token compare mask, a 16 bit tokenhold length register, and a single token priority enable bit. The tokenenable bit controls whether the token passing scheme is active in switchmode. If the enable bit is set to 1, the token passing scheme is enabledfor the switch chip. All switch chips must set their token enable bit tothe same value before transitioning to switch mode.

[0212] This mechanism allows for fairness to be maintained throughoutthe system. This feature is critical in congested server environmentswhere starvation of disk access can freeze an application. Thecirculating fairness token prevents this from happening.

[0213] The token position counter identifies the port or port group thatholds the token. A port holds the token if the port ID of the portAND²-ed with the 6 bit compare mask matches the token counter valueAND-ed with the compare mask. If a port holds the token, its requestsare made with high priority. Ports wishing to request a port that holdsthe token are denied so that the camp list of the port holding the tokencan drain and local requests can be serviced.

[0214] The token position counter is incremented when amanagement-specified number of wall clock counter bits roll over tozero. The wall clock counter is a 24 bit counter which is incrementedevery word clock period of 40 ns, resulting in a range of approximately⅔ of a second. The token position increment signal is computed OR-ingthe lower 8 bits of the wall clock together with the bits resulting froman AND operation between the upper 16 bits of the wall clock and a 16bit token length value, which should be set to a string of logic onespadded to the left with logic zero. When the resulting value is zero,the token counter is incremented.

[0215] The length of the string of logic ones in the token lengthregister determines the amount of time that each port will be assignedthe token. For example, if the value of the token length register is 0¥3f, the token counter will be incremented every time the lower 14 bits (8LSB's+6 additional bits) are zero, or every 2¹⁴*40 or about every 650usec.

[0216] Camping on a port means that a remote request waits on a busydestination port until that port becomes available. A non-camped requestwill simply be closed at the source if the destination port is busy.Campers are ordered in a camp list FIFO which has a depth of 8 entries.

[0217] Low and high priority are allowed to camp on a port if the camplist for that port is not currently full and the desired port is notholding the token. High priority requests differ only in that at thedestination port they may be queued up to the depth of the camp list,while low priority requests queue to the low-priority camp depth set bymanagement. All requests are serviced by a single queue with FIFOdiscipline. An alternative embodiment uses separate queues for high andlow priority thus allowing high priority to jump to the head of theline.

[0218] Camping on a port that is already camping on another port cancreate a deadlock situation in which two or more camped ports form acycle. No forward progress can be made when this occurs, resulting in a“fatal embrace”. Such deadlocks are prevented by scoreboarding all portswhich have outstanding requests. If the request has been observed on theprotocol bus but no response has been issued by the target port, thenthe requestor port is marked as camped in the scoreboard. All requeststo ports marked in the scoreboard as camped are immediately denied. Oncethe port in question has been issued a response, implying a grant and anallocated backplane channel, the formerly camped port can beginaccepting campers.

[0219] One anomaly exists which involves no-priority campers. Due toimplementation issues, it is easier to permit the very first request tocamp regardless of priority. Thus, even a no-priority request, if issuedto a port that is busy and has an empty set of camp lists, will becamped. This anomaly occurs because there are difficulties in detectinga busy port unless the port is involved in a remote conversation thatcan be reflected in the scoreboards.

[0220] Transitions from Hub to Switch

[0221] In order to transition the system from switch mode from hub mode,the system must first be locked. Before initiating a lock from hub mode,the management software should set all parameters necessary for switchmode transition while the system is in hub mode so that the time spentduring the transition in the locked state is minimized. Sincetransactions cannot be initiated while the system is locked, it ispossible that an NL_port could time out and LIP (initialize) if thesystem is locked for too long, causing the hub to switch transition tofail.

[0222] Once the system is locked, external logic asserts a signal whichsets the port multiplexers such that each port is partititioned from thesystem. During this time, each loop is broken, and IDLEs are driven ontothe local loops in order to nullify ARBs that may still be present fromhub mode arbitration. Next, the locking signal is deasserted, and, aftera brief delay in which IDLEs are sent to the outbound port, theswitching chips transition to the monitoring state and switch modeoperation commences.

[0223] If a LIP or other exceptional condition is detected during theswitch mode transition, the system sets an output to a state thatindicates the mode transition failed and the system is still in hubmode.

[0224] Destination Node Address Discovery Processes

[0225] The associations between node addresses (AL_PAs) and theirassociated ports are determined at initialization time through alearning process referred to herein as OPN trapping. An OPN primitive issaid to be trapped in hub mode when it has been transmitted onto a localFCAL net and an RRDY primitive received in its place. When an OPN hasbeen trapped, the switch chip has the destination node address (AL_PD)of the device residing on the FCAL net where the trap occurred as wellas the port ID of the port servicing the loop. This information isstored in the LUT routing table 228.

[0226] Storing addresses into a single LUT is not sufficient for properswitch operation. The data in the LUT routing table of each switch chipin the system should be filled in before transitioning to switch mode.This can be achieved in two ways. The slower method, yet simpler toimplement in the circuitry of the system, is to use the managementinterface to read every address in every LUT and build the LUT contentsin software. The CAMs of every switch chip are then loaded through themanagement interface.

[0227] The CAMs can also be synchronized as the OPNs are trapped throughcommunications on the protocol bus. Since the protocol bus is onlyutilized for scoreboarding during switch mode, every trapped OPN resultsin a protocol bus transaction that advertises the node address and theport ID to all switch chips.

[0228] The discovery of every utilized AL_PA through OPN trapping can befacilitated in a passive or an active manner. During hub mode, ifpassive OPN trapping is enabled, normal traffic will result in trappedOPNs which will eventually populate every LUT with the address-portassociations of every NL_port in the system.

[0229] AL_PA active discovery is a procedure by which the switch chiplearning process is acelerated by additional logic on each switch chip.Active discovery is initiated by asserting a signal on each switch chipwhile it is in locked state. Once active discovery is initiated, asignal BCST_BUSY is driven low, and a 10-bit counter cycles through the1024 10-bit words. Each word is passed through an address encoder whichgenerates a signal that is active whenever the 10-bit word correspondsto a legal AL_PA. For each such word, the discovery process is triggeredfor each port on the switch.

[0230] The active discovery process starts with an OPN transmitted ontothe local loop using the counter word as the AL_PD or destinationaddress. The OPN is immediately followed by an interframe gap and a CLS.This sequence will travel around the loop and return to the discovererif a device with that particular AL_PD is not on the local loop. If thenode with that destination address is on the local loop and the node isnot bypassed, the node will consume the OPN and CLS and send a CLS ofits own. In this case, the discovery logic will write the AL_PA to portassociated into the CAM. After an OPN or CLS is received at the receiveport of the port which launched the OPN onto the local loop, the activediscovery state machine pauses until the counter reaches another validAL_PA. Once all valid addresses are attempted, the entire ArbitratedLoop address space has been deterministically mapped. This causes theBCST_BUSY signal to be allowed to float to logic 1, signalling thatactive discovery is complete. Active discovery may be aborted bydeasserting the signal which initiated it which results in immediatereset of the discovery state machine.

[0231] Loop Port State Machine Policy

[0232] The loop port state machine is not a strict implementation of aconventional FCAL loop port state machine since it must switch OPNsequences and subsequent frames to and from the backplane and postmessages to and read messages from the protocol bus and update thescoreboard and carry out the learning protocols.

[0233] While the exact design of the state machine is not critical tothe invention, all designs must be able to perform the functionsdescribed above for at least one embodiment and, at least in thepreferred embodiment, must adhere the following rules.

[0234] 1) The LPSM must preserve word alignment unless it is absolutelyimpossible to do so. Since bugs exist in the HP Tachyon™ design whichfill the loop with nonsensical transmissions if a K28.5 is followed by acharacter and another K28.5, any LPSM implementation that shares theloop with a Tachyon-based device must exercise great care in aligningoutput words.

[0235] 2) The LPSM must not disturb OPN-CLS or SOF-EOF symmetry, norshould it unintentionally disturb credit flow by removing and addingRRDYs. In dual simplex cases, the LPSM will intentionally manipulatecredit by absorbing RRDYs and rerouting them to an alternatedestination.

[0236] 3) The LPSM must conform to the FCAL requirements regardinginterframe gaps. Also, the LPSM should avoid interframe gap conditionswhich are FCAL legal, but cause interoperability problems with currentequipment.

[0237] 4) The LPSM may operate in unfair mode in order to prioritizeremote connections over local connections, but it must not breakarbitration by prematurely resetting the access window, or starvationmay occur.

[0238] 5) The LPSM must allow loop initialization to be triggered and toproceed in a reasonably normal manner.

[0239]FIG. 11 is state table describing the various LPSM states in thepreferred embodiment.

[0240] Loop Port Unfairness

[0241] An LPSM always operates in unfair access mode in order to obtainthe loop as soon as possible. The current arbitration window is notobserved and the high priority ARB (usually ARB(0), but programmable bythe management software to be any AL_PA) is used to gain control of theloop. Simply put, the loop port state machine issues ARB on the loopwhenever a remote request is pending and continues to do so until thesame ARB is received back at the LPSM indicating that the LPSM has wonarbitration.

[0242] If the LPSM receives an OPN or RRDY primitive, then anotherdevice on the loop won arbitration. In this case, the LPSM continues toissue its ARB, and monitors the loop until arbitration is won. Whilearbitrating, an OPN may arrive from the loop; the AL_PD of the OPN isthen used to search the LUT to determine whether the OPN is to a remoteor local device. If the OPN is remote, the LPSM synthesizes a CLS in anattempt to end the tenancy of the device on the local loop which sentthe OPN which is interfering with the arbitration attempt of the LPSM.Thus, remote tenancies which have already traversed the protocol bus andare present camped are heavily favored over local devices who areattempting to make a remote connection. If however the OPN received fromthe local loop during arbitration by the LPSM is directed to a localdevice, the LPSM continues to arbitrate while processing the localtransaction normally.

[0243] Interframe Gaps

[0244] The ports on every switch actively participate on their loopswhen they are facilitating a remote tenancy. The two primitives RRDY andCLS are used according to FCAL net interframe gap policy. The specificpolicy that the switch chips use is that if a current primitive is afill word and two consecutive fill words have previously been detected,then insertion of the RRDY or CLS may take place. For purposes of thispolicy, fill words are defined as ARB(x) and IDLE primitives.

[0245] Fill Word Insertion

[0246] Fill words are generated and inserted onto the local loop whenthe LPSM is not in the LPSM_MON state (hereafter the states of the LPSMwill be referred to by the acronym that follows LPSM_ in FIG. 11). Rulesgoverning fill word generation ensure that the current arbitration stateof each loop is maintained independently. The problem for each switchchip when it is coupled to a source loop, i.e., the loop having thesource node, is that it must send fill words that are relativelyinnocuous. An NL_port in an OPEN state will transmit ARB(FO) which ituses to determine if any other ports on the loop are currentlyparticipating in arbitration. If the open NL_port receives ARB(FO) inreturn, the NL_port has the option of retaining the loop and sets itcurrent fill word to IDLE, thereby resetting the arbitration interval.The LPSM needs to send a management programmable low priority ARB thatdoes not possess the potential to disturb the local loop access statelike ARB(FO) does. For this purpose, the ARB(F7) is suitable, as it isonly used by a non-participating port to quiesce the loop prior tosending a loop initialization primitive LIP in order to receive a validAL_PA.

[0247] The fill word generation matrix for the source loop is given inFIG. 12, and the destination fill word generation matrix is given inFIG. 13. In the source loop LPSM, the ARB(F7) state is entered when alocal OPN is received, and a minimum of 6 current fill word primitivesare sent when relinquishing the loop. In the destinaton loop LPSM, theARB(F0) state is entered when a remote connection request is received,and the ARB(F0) state is entered when a tenancy is established and anOPN is passed. A minimum of 6 current fill words are sent whenrelinquishing the loop.

[0248] Remote Data Transfer

[0249] In order to reduce the amount of decoding required to merge thedata stream of a remote connection into the flow of data on the localloop, very few primitives are passed across the backplane.

[0250] Miscellaneous Primitive Handling

[0251] The most common primitives that the switch chips must deal withare: IDLE, RRDY, ARB, OPN, SOF, EOF, CLS and LIP. The other primitivesthat may be encountered include: NOS, OLS, LR, LRR, MRK, LPB AND DHD.The handling of these other primitives is described below.

[0252] NOS, OLS, LR and LRR Primitives

[0253] When NOS or OLS primitives are detected, on the inbound port of aloop, the OLD_PORT detect flag of the port is set so that the managementsoftware can detect the condition. LR and LRR primitives will be fedback onto the local loop, but not through the backplane.

[0254] MRK Primitive

[0255] The MRK primitive is discarded if issued during a remote tenancyin dual-simplex. Otherwise, MRK primitives are passed around the localloop and across any remote connection. However, it is unlikely that aMRK will find its destination unless it is issued inside of a tenancy inwhich the target is on the destination loop.

[0256] LPB and LPE Primitives

[0257] The LPB and LPE primitives are trapped and written to managementsoftware. A detection flag is set for either an LPB or an LPE, and theAL_PD and AL_PS are written to a 20-bit register that is accessible bythe management software.

[0258] DHD Primitives

[0259] In dual simplex mode, all dynamic half duplex primitives aretranslated into CLS primitives. Otherwise, they are passed unaltered totheir destination.

[0260] Protocol Bus Definition

[0261] The protocol bus is the medium by which the scoreboards for eachswitch chip are kep current. It also serves to communicate switchconnection requests and responses between switch ports. The protocol busis defined as an 18 bit bidirectional data bus named PBD, a requestoutput PBREQ0 for each switch chip, a grant input named PGGRNT1 to eachswitch chip, a shared bus idle input signal PBIDLE1, and shared frameavailable input signals named PBFRM1. The protocol bus runsasynchronously with respect to the 106.25 MHz core circuitry to which itinterfaces. It is only necessary to provide a clock of less than 50 MHzfor the bus to function properly.

[0262] When a switch chip wishes to transmit a data frame on theprotocol bus, it drives its PBREQ0 output. The PBREQ0 of each switchchip in the system is fed into external arbitration logic whichprioritizes the inputs and asserts a single PBGRNT1 output in the samecycle. The chip that receives the asserted PBGRNT1 deasserts PBREQ0 andthen drives the PBD bus in the next cycle. The arbiter asserts PBFRM1 inthe cycle following any PBGRNT1 to notify each switch chip that a dataframe is present on the protocol bus.

[0263] Once a switch chip obtains a PBGRNT1, it must not assert PBREQ0until it sees PBIDLE1 asserted from the arbiter. The PBIDLE1 is assertedwhenever no chip is requesting the bus in a cycle, indicating that theprotocol bus arbitration window may be reset.

[0264] Each protocol bus data frame possesses a slightly differentformat. These formats are depicted in FIGS. 14A through 14E. Theabbreviations BP CHNL in FIGS. 14B and 14C stand for backplane channel.The abbreviation PID in FIG. 14E stands for port ID.

[0265] The connection request data frame of FIG. 14A is sent by a portthat wishes to initiate a connection to a remote destination port andhas found that the destination camp list can accept a request of thedesired priority, and that the destination port is not currentrequesting a destination for itself. When a connection request isbroadcast, every switch chip marks its 64 bit requester scoreboard andincrements its camp list depth scoreboard for the destination andpriority.

[0266] Chip remote request frames have a 0x0 pattern in the LCL fieldshown at 300 in FIG. 14A while chip local requests place a 0x3 in thisfield. This difference in format allows switch chips to update channelallocation scoreboards only on chip remote requests. If a chip remoterequest hits a channel that has been placed in an age list, a channelidle scoreboard for the channel owned by the destination port iscleared; otherwise, a need counter is incremented as will be explainedin the next section.

[0267] A connection response having the format of FIG. 14B is broadcastby a port that has serviced a camp list entry, and indicates that aconnection may be initiated over a backplane data channel specified infield 302. A connection response results in the clearing of therequestor scoreboard for the requester ID given in field 304 of theresponse frame, and it also results in setting the responder scoreboardfor the destination port. The port that own the allocated channel andthe channel number are entered into the age list on the initialallocation of the channel. A responder port that resides on the sameswitch chip as the requestor, indicating a chip-local connection, muststill broadcast a response frame so that the responder scoreboard isupdated in all the other switch chips. This type of chip-local connectrespose carries an invalid channel ID code of 0xf, which indicates toreceiving switch chips that the backplane usage scoreboard should not bemodified.

[0268] A responder channel idle frame having the format of FIG. 14C isdriven on the protocol bus when a backplane channel completes atransaction. In the default system configuration, the channel isretained by the responder until the channel is required by anotherresponder, thus reducing setup overhead if the destination node thatjust finished a tenancy over the channel is a frequently accesseddestination. When an idle frame is received, each switch chip updatesits channel idle scoreboard to indicate the channel is available.

[0269] The no-op frame of FIG. 14D is used as a late abort of a responsedue to channel allocation. This may occur when a request has been queuedfor transmission but a conflicting request is received before the queuedrequest can be transmitted. The queued request is checked a final timebefore transmission, and if a local deny must take place, the frame isconverted to a no-op and sent to get it out of the queue. Any other wayof purging the queue will also suffice.

[0270] The lookup table update frame of FIG. 14E is transmitted when anOPN is trapped on a local loop in hub or locked mode. The frame is thenused by each switch chip to enter a node address AL_PA equal to thecontent of field into its LUT with a port ID mapping as specified infield 308.

[0271] Backplane Channel Allocation

[0272] The example given herein for the configuration of the backplanehas 14 separate backplane data channels, three of which may be dedicatedto other uses such as broadcast. Channels are allocated by destinationports and remain assigned to them for as long as possible. A channelscoreboard indicates if any channels are free and is used to hold offany pending responses from ports which are not already connected to adata channel. If this is not the case, a need counter is incremented.When this need counter exceeds the physical number of data channels,then the backplane attempts to free up a channel while the destinationport attempts to obtain a connection grant response from its LPSM. Alldata channel connections (identified by response frames on the protocolbus) are stored in the age list which indicates the channels that havebeen held the longest and the port that currently own those channels.When the need counter exceeds the number of data channels, each switchchip consults its age list and selectes the channel to be relinquished.Idle channels are broadcast on the protocol bus using the frame of FIG.17 when the camp list of the port owning the channel empties completely.The oldest owned channel is freed by the switch chips that owns thechannel and all switch chips update their scoreboards to reflect the newstate. For every channel that is freed, the need count is decremented byone.

[0273] Channels can also be freed in blocks of size greater than one.The size of the block of channels freed is determined bymanagement-programmable parameter. The switch chip management logic canalso instruct the backplane to always free a channel when it becomesidle, rather than only freeing the channel when the need arises.

[0274] Switch Fairness

[0275] Starvation is a problem because most of the fairness policiesimplemented by the switch chips guarantee remote access fairness toswitch ports and not to the individual NL_ports. Thus, while an NL_portis guaranteed access to its local loop and is guaranteed to winarbitration within one FCAL net access window, its tenancy will not besuccessful if the NL_port attempts to transmit to a remote port and isdenied connection by the switch. Usually, randomness of access patternswill result in statistical fairnesss. However, it is not uncommon toencounter degenerate cases where this behavior can occur repeatedly on agiven NL_port, resulting in starvation of the port.

[0276] To improve the fairness of access to the switch at the NL_portgranularity level, it is necessary to guarantee that all NL_ports on alocal loop get some fraction of the switch bandwidth. Mechanisms existfor escalating from no-priority requests to low-priority requests basedon the number of consecutive denials of access issued by a remote portstate machine and for rotating a fairness token among all the ports toguarantee that each port gets a turn at high priority access.

[0277] Broadcast Support

[0278] In the preferred embodiment, each switch chip's LPSM includeslogic to support three possible broadcast modes: that of the broadcastsender, the broadcast server transceiver, and the broadcast receiver.For a switch chip to be able to send broadcasts, it must include logicto decode a broadcast sequence as described below, logic to determinewhether the broadcast channel, if it exists, is busy, logic to forward abroadcast sequence directly to the local loop as it is received, andlogic to write a preamble to the broadcast sequence and place thesequence on the broadcast channel if it is available.

[0279] To decode a broadcast sequence, each switch chip should have thethe following structure. Broadcast sequences are detected on the inboundport from the local loop. A broadcast sequence is defined to be thesequence of one or more broadcast OPN groups terminated by a CLSprimitive. A broadcast OPN group is defined to be one or more selectiveor broadcast OPNs followed by data frames.

[0280] The LPSM always forwards a broadcast OPN group back to its localloop, guaranteeing that all local NL_ports see the broadcast and thatthe sender of the broadcast can clean up the local loop. If broadcastmode is enabled by the management software, the switch port alsoattempts to forward the broadcast to the predefined backplane broadcastchannel. If the channel is not already busy with a broadcast that isalready in flight, the switch chip will prepend a unique, identifiersequence to the broadcast sequence and transmit the prepended sequenceonto the broadcast channel. During the transmission, any broadcasts onthe channel but upstream from the broadcasting switch chip will bediscarded. As a result, only one broadcast may reach the broadcastserver. When the server receives a broadcast, it reads the broadcasterID prepended to the sequence. This identification mechanism excludes theoriginal broadcaster from transmitting data onto the local loop a secondtime. The broadcast server then sends unicast to each of thedestinations on the broadcast list.

[0281] The server comprises one port of a switch chip that is configuredto only listen and transmit on the broadcast channel, and a broadcastserver board that is connected to the switch chip by its serial loopport. Broadcast sequences are received by the dedicated broadcast portand forwarded to the switch port connected to the broadcast server boardfor storage.

[0282] In order for a switch chip to be able to receive broadcasts, theLPSM must include logic that discriminates whether an incoming broadcastoriginated at the port or at some other port from the data in thepreamble prepended to the broadcast sequence.

[0283] In order for a switch chip to be able to function as thetransceiver interface to the broadcast server, a switch port must beconfigurable so that it only forwards data from the broadcast channel tothe loop port and forwards data from the loop port back to the broadcastchannel. This port should not be effected by state change requests orother exceptional conditions if it is configured as a broadcasttransceiver.

[0284] Buffered FCAL Switch

[0285] Referring to FIG. 29 there is shown a species of a buffered FCALswitch which falls within a separate second genus of FCAL switches,suitable for Class 3 Fibre Channel operation only. Switches in thisgenus still uses the destination address in the OPN to find the remoteport but uses buffers instead of hold back flow control to complete thetransaction to busy remote ports. Specifically, species within thisgenus will use the destination address of the OPN from the source nodeto find the location of the remote port. The destination address is usedto look up the port ID of the switch port couped to the destinationaddress. Suppose Then the status of that port will be checked. If thestatus is available, a connection request will cause a connection to beset up between the source node and the destination node via a sourceport connected to the source node and a destination port connected tothe destination node. The buffer comes into play when the destinationport is busy. In this situation, in the first genus described above, thenormal primitives of the FCAL protocol are used for flow control toprevent the source node from transmitting any frames of data until thedestination port becomes available. In the second genus defined in thisparagraph. a buffer big enough to hold one or more complete frames ofdata is included in the front end of each switch chip, or multiplebuffers each big enough to store a frame of data are included with eachswitch chip front end. Each of these buffers will serve as an auxiliaryswitch port and have its own connection to the backplane in some speciesor a single shared connection to the backplane through a multiplexer canbe used. The preferred species uses multiple buffers each with its ownconnection to the backplane in addition to a connection directly fromthe switch port to the backplane for direct connections withoutbuffering. In some species, a single shared buffer or multiple sharedbuffers on the backplane or in some central location may be used.

[0286] In this second genus, the way the buffers are used is for thesource port to generate an RRDY sua sponte when it finds from a check ofthe scoreboard that the destination port is busy. The RRDY is sent tothe source node and causes it to output a frame of data. This frame ofdata is stored in the switch port's buffer. Then a message is sent tothe destination port indicating that the auxiliary buffer of the switchport is holding a frame of data for the destination port. This auxiliarybuffer ID is added to the camp list for the destination port. When thedestination port becomes available, a message is sent back on theprotocol bus indicating that the destination port is now available andnaming the backplane channel to use. A connection through the backplaneis then established to this channel by the auxiliary buffer connectioncircuitry and the destination port, and the data in the auxiliary bufferis transmitted. If the switch port has multiple auxiliary buffers, theyeach have their own IDs and, preferably, each has its own switchingcircuitry to make a connection to the backplane.

[0287] In this second genus, each auxiliary buffer has circuitry coupledto the return path to recognize RRDYs transmitted back by thedestination node and to count them (or store them) and to wait for aconnection between the source port and the RRDY counting circuit if theconnection is not continuous such as in some cases where multiplebuffers are present in each switch port. These stored RRDYs (or selfgenerated in the case of a count only) can be transmitted to the sourcenode in the case of full duplex or mixed with frames from a third nodein the case of a dual simplex connection and transmitted to the sourcenode. Each source port also has shared circuitry for each FCAL net whichrecognizes incoming RRDYs from the source node and counts them or storesthem. These source node generate RRDYs can be transmitted to thedestination node in the case of full duplex or transmitted to a thirdnode in the case of dual simplex.

[0288] The operation of such a switch is now described with morespecificity in connection with the species shown in FIG. 29. We firstconsider a full duplex transaction. In the species of FIG. 29, anOPN/RRDY detector and RRDY generator circuit 450 detects any incomingOPN or RRDY primitives coming in from the source node 452 on the localFCAL net 454. The destination address of the OPN is latched and sent tothe lookup table circuitry represented by block 456 via line . Thelookup table looks up the ID of the port connected to the destinationnode and determines its status. Suppose a first OPN with a destinationaddress for node 451 is received from source node 452 and the status ofdestination node 451 and its switch port 453 is available. In this case,protocol bus interface circuitry in block 456 sends a connection requestvia protocol bus 458 to protocol bus interface circuitry in block 457 ofswitch port 453 requesting a connection. This results in a connectresponse message naming the backplane channel to use. The loop portstate machines in blocks 456 and 457 then send commands via buses 466and 467, respectively, to their respective port multiplexers 460 and 459to establish connections between “straight through” data paths 462 and464 to the designated backplane channel in backplane 465. The LPSM inblock 456 also controls switches 468 and 470 to make connections towires 462 and 464, and the LPSM in block 457 control switches 469 and471 to make connections to “straight through lines 473 and 475.Destination node responds with an RRDY and that gets passed to sourcenode 452 through the connection just described and the transactionproceeds normally until completed and the backplane channel isrelinquished.

[0289] Now suppose switch port 453 was busy. In this scenario, thelookup process using the destination address in the OPN determines thatan immediate connection is not possible. In this case, the LPSM in block456 controls switch 470 to make an outbound connection to buffer 1A viawire 472 and controls switch 468 to make an inbound connection via wire474. The LPSM then commands switch 476 to open long enough for an RRDYto be sent to source node 452 and commands RRDY generator circuit 450 togenerate an RRDY on line 478 and send it to source node 452. The RRDYcauses source node 452 to output a frame of data. This frame passesthrough line 480, SERDES 482, switch 470 and line 472 into buffer 1A andis stored there. The circuit 450 then generates a CLS and sends it tosource node 452. Then switch 476 is closed. LPSM in block 456 then sendsa message on protocol bus 458 to the protocol bus interface circuitryand LPSM in block 457 that it has a frame of data waiting fordestination node 451. This message gives the ID for the buffer 1A andcauses that ID to be put on a camp list for destination node 451. Whendestination node 451 becomes available, LPSM in block 457 sends amessage back to LPSM in block 456 saying “send data in buffer 1A onbackplane channel X”. The LPSM in block 456 then controls portmultiplexer 460 to establish a connection between wires 484 and 486 tothe designated backplane channel. LPSM in block 458 controls switchs 469and 471 to establish connections to wires 473 and 475 and controls portmultiplexer 459 to connect wires 473 and 475 to the designated backplanechannel. The LPSM in block 456 then causes a switch 490 to close andcauses Buffer 1A to output its frame of data onto the backplane channelwhere it gets transmitted to the destination node 451 via straightthrough wires 473 and 475.

[0290] In some embodiments, the circuit 450 will not send a CLS to thesource node 452 after it sends a frame of data into the buffer 1A. Inthese embodiments, the connections will be maintained so that if thereis more than one frame, it can be sent as soon as the destination nodebecomes free by a straight through connection. In these embodiments, theLPSM in block 456 also causes switch 492 to close for a return path andcauses switch 468 to make a connection to return path wire 474 so anyRRDYs output by the destination node are transmitted to the source node452 once the destination node becomes available. The LPSM also thencauses switch 470 to make a connection between outbound path 480 fromsource node 452 and “straight through” outbound wire 464 so subsequentframes can be sent straight through. The transaction then completes as astraight through transaction. Buffers in the switch ports such asbuffers 1A and 1B may be big enough to hold more than one frame in somespecies to avoid having to establish a separate tenancy for each frame.

[0291] Since the preferred method in the buffered switch is to close thesource node after it outputs one frame into the buffer, it is then freeto generate new OPNs to other destination nodes. Those OPNs can causeeither straight through or buffered connections to their destinationsdepending upon the status of the destination port. The LPSMs cancooperate after the table lookup using the new OPN destination addressto control the switches 468 and 470 and their counterparts in thedestination port to establish a straight through connection to the newdestination via another backplane channel. In the meantime, the buffer1A and its associated switches and the LPSM are cooperating to act as anindependent switch port such that when the destination node for the datain the buffer becomes available and a connect response message isreceived, it can independently send its data to the destination node thedifferent backplane channel assigned in the connect request message.This increases throughput since the same switch port may besimultaneously be sending data to two or more different destinations.Buffered FCAL switches such as that shown in FIG. 29 have the additionaladvantage in that it prevents source blocking. The source may downloadone or more frames into the buffer for a destination node and then moveon to its next transaction thereby reducing or eliminating head endblocking.

[0292] Dual simplex is also possible to increase throughput further byusing circuit 450 to count RRDYs emitted by the source node and usingLPSM in block 456 to award those RRDYs to a third node. This way, theRRDYs will cause the third node to send data inbound to the source nodevia a separate backchannel connection with LPSM in block 456 controllingswitch 468 to establish the inbound path to source node 452 via thebackplane channel assigned by the third node and straight through wire462. Outbound data from the source node to a destination node can besent by straight through wire 464 or through one of the buffers. RRDYsemitted by the destination node are received by the destination port andsent via the protocol bus to the LPSM of the source port or a message issent each time an RRDY is received from the destination node by thedestination port informing the source port of this fact. The source portLPSM then controls circuit 450 to generate an equivalent number of RRDYsand mix them in with the data frames from the third node on the inboundpath 492 to the source node 452 to keep the source node outputting dataframes destined for the destination node.

[0293] The RRDY capture circuits are used in species where RRDYs areemitted by the destination node, but the switches 468 and 470 are instates such that the RRDYs cannot be immediately be sent to thedestination node such as where the source port is generating multipleOPNs to different destinations and filling up all its buffers one byone. In these embodiments, the nodes will have to be non standard inthat they will have to OPN a destination, download one or more framesinto a buffer, receive a CLS from circuit 450, OPN a new destination,download another one or more frames into another buffer, receive anotherCLS from circuit 450 and continue this process until all buffers havebeen used. The LPSM will commutate the switches to make successiveconnections to the buffers and straight through connections as neededand keep cycling through these connections. The RRDY capture circuitswill count the number or RRDYs received or emitted by the destinationnodes, and when the switches 468 and 470 and 490 and 492 are again inposition for communication with the destination node, the source nodewill open itself spontaneously for that destination, receive any storedRRDYs from that destination and send an appropriate number of frames tothat destination either by a straight through connection or a bufferedconnection.

[0294] Fairness is implemented in the species within the second genus inthe same way as in the species of the first genus. A fairness token iscirculated, and when any switch port has the token, it assumes thehighest priority. The fairness token can be circulated amond the switchports on a separate fairness token bus (not shown in FIG. 29) or via theprotocol bus which is the embodiment symbolized by FIG. 29.

[0295] Although not shown for simplicity in FIG. 29, each switch portalso includes a local loop bypass data path to keep purely localtransactions confined to the local FCAL net. Note also that althougheach FCAL net is FIG. 29 is shown as having only one node, multiplenodes on each FCAL net are also possible. Note also that although FIG.29 shows only one switch port per switch chip, multiple switch ports perswitch chip are also possible similar to the structure shown in FIG. 7but using the architecture of FIG. 29 for each switch port.

[0296] Appendix B attached is the UUencoded Verilog description of thepreferred embodiment of the entire switch chip integrated circuit. Atthe end of Appendix B is the C language source code for the UUencodingand decoding program to enable decoding of the Verilog and documentationfor using the UUencoding and decoding program.

[0297] Although the invention has been described in terms of thepreferred and alternative embodiments disclosed herein, those skilled inthe art will appreciate numerous modifications that can be made. Allsuch modifications and alternatives are intended to be included withinthe scope of the claims appended hereto.

What is claimed is:
 1. A bufferless switch for coupling to a pluralityof FCAL nets and having a crossbar switch and FCAL loop interface portcircuits structured to use the OPN and RRDY primitives of the FCALprotocol for hold back flow control to eliminate the need for a bufferand said ports and crossbar switch structured to provide multiplesimultaneous loop tenancies.
 2. A bufferless switch for coupling to aplurality of FCAL nets and having a crossbar switch and FCAL loopinterface port circuits structured to use the OPN and RRDY primitives ofthe FCAL protocol for hold back flow control to eliminate the need for abuffer and said ports and crossbar switch structured to provide dualsimplex communication across said switch.
 3. A bufferless switch forcoupling to a plurality of FCAL nets and having a crossbar switch andFCAL loop interface port circuits structured to use the OPN and RRDYprimitives of the FCAL protocol for hold back flow control to eliminatethe need for a buffer and said ports and crossbar switch structured toprovide multiple simultaneous loop tenancies and dual simplexcommunication across said switch, each said port including a localbypass data path which can be switched to keep primitives and dataframes generated on a local FCAL net coupled to said port routed so asto stay on said local FCAL net such that local loop tenancies on eachlocal FCAL net coupled to a port which do not involve transmission ofdata or primitives from one port to another can occur simultaneously,and each port including local learning memory means for storing arouting table and scoreboard table, and functioning to fully cache the8-bit addresses of all nodes on each said local FCAL net coupled to aport along with the ID of the port coupled to each node in said routingtable and the busy, available or no privilege status of each other portand whether each busy port is available for dual simplex communicationsin said scoreboard table, each port structured to write the contents ofeach said routing table and scoreboard table by learning either fromwatching local traffic in said port bound for local destination nodes orremote traffic leaving said port and crossing said crossbar switch boundfor remote destination nodes or by conducting an active discoveryprocess, each port structured to write the contents of said scoreboardtable by learning from watching messages posted on a protocol bus, eachsaid port structured to use destination addresses in OPN primitivestransmitted by nodes on said local FCAL net as search keys to searchsaid routing table to determine whether the destination node is local orremote and to which port it is connected and, if local, to switch saidlocal bypass to keep the loop tenancy confined to said local FCAL net,or, if remote, to generate a connection request message to said portcoupled to said destination node, each said port being structured toreceive such connection requests and respond by arbitrating for controlof its local FCAL net and pick a data path through said crossbar switchand generate a reply message naming the data path through said crossbarswitch to use in exchanging data frames and primitives, each said portalso being structured to respond to receipt or generation of said replymessage by generating commands to establish said data path named in saidreply message through said crossbar switch, each said port alsostructured to circulate a fairness token on a fairness token bus coupledto all such ports and to used said fairness token to increase itspriority level of access privilege to busy nodes when the fairness tokenis in possession of the port so that no port can be starved, and then tocirculate said fairness token to a neighboring port such that all portseventually get said fairness token.
 4. The apparatus of claim 1 whereinsaid port circuits are each integrated on a single integrated circuitwith a portion of said crossbar switch.
 5. The apparatus of claim 2wherein said port circuits are each integrated on a single integratedcircuit with a portion of said crossbar switch.
 6. The apparatus ofclaim 3 wherein said port circuits are each integrated on a singleintegrated circuit with a portion of said crossbar switch.
 7. A switchfor selectively, concurrently coupling the nodes on a plurality of aFibre Channel Arbitrated Loops together comprising: a multiplexed bus; acrossbar switch; a plurality of switch control means, each for couplingto one local FCAL network, and each coupled to said multiplexed bus andcoupled to said crossbar switch, each for receiving OPN primitives froma source node on the local FCAL network to which it is coupled anddetermining if the destination node identified in said OPN is on saidlocal FCAL network, and, if so, routing the OPN to the destination nodevia the local FCAL network, but if the destination node is not on saidlocal FCAL, transmitting a destination node location request and thedestination address of the destination node identified in said OPNprimitive to all said other switch control means via a channel on saidmultiplexed bus, and for scanning messages received back from said otherswitch control means via a channel on said multiplexed bus to determinewhich switch control means and local FCAL network to which flow controlprimitives and data are to be transferred to accomplish the desired datatransfer, and when a grant message is received from another switchcontrol means indicating said other switch control means is the place tosend flow control primitives and data and indicating that the FCAL towhich said other switch control means is coupled is available for a looptenancy, for cooperating with said other switch control means toestablish a data transfer path across said crossbar switch, and usingsaid data transfer path to transfer flow control primitives between saidsource and destination nodes, and each switch control means also forreceiving destination node location requests and destination addresses,and using said destination address to search a table of destinationaddresses of nodes on the local FCAL of said switch control means todetermine if the destination node identified in said destination nodelocation request is on the local FCAL to which said switch control meansis coupled, and, if so, for arbitrating for control of said local FCALand, when control is won, sending a grant message back to said switchcontrol means which initiated said destination node location requestidentifying the switch control means coupled to the local FCAL on whichsaid destination node is resident and for sending one or more messagesto said switch control means which initiated said destination nodelocation request for purposes of controlling establishment of said datatransfer path across said crossbar switch.
 8. A packet switching switchfor coupling a plurality of Fibre Channel NL nodes and/or Fibre ChannelArbitrated Loop network (herafter FCAL nets) together to all concurrentdata exchanges between a plurality of pairs comprised of one NL node orFCAL net and one other NL node or FCAL net, comprising: a crossbarswitch; a protocol bus; a plurality of port circuits each having aninput and an output for connecting to an FCAL net coupled to one or moreNL nodes and capable of implementing a Fibre Channel loop protocol, andeach having a crossbar switch port coupled to said crossbar switch;means coupled to said protocol bus for maintaining a scoreboard tablecontaining at least status information and a routing table eithercentrally located or in each port circuit, said routing table containingdata mapping destination addresses of NL nodes to port IDs; and whereinsaid port circuits are coupled to said means for maintaining ascoreboard table and routing table and wherein said port circuitsfunction to establish connections between themselves by usingdestination addresses in OPN primitives received from source nodes tosearch said routing table to determine the ID of a remote port coupledto the destination node having the destination address in the OPN and,using that port ID to search said scoreboard table to determine statusof the remote port, and then exchanging messages with said remote portto cause it to arbitrate for and take control of its local FCAL net andestablish a particular channel through said crossbar switch and use saidchannel to transmit primitives and data frames between said source nodeand said destination node.
 9. The apparatus of claim 8 where said switchis structured to be capable of operating at high throughput rate, butneither said crossbar switch nor said port circuits have the amount ofmemory that would be needed in a fabric switch to operate at the samethroughput rate.
 10. The apparatus of claim 8 wherein said port circuitsare not FL_ports.
 11. The apparatus of claim 8 wherein said portcircuits have no error recovery protocols and are structured to use holdback flow control such that frames are not transmitted until theconnection is established through said crossbar switch to thedestination node and is ready to stream the data directly from thesource node to the destination node without ever buffering the data andnever dropping a frame.
 12. The apparatus of claim 8 wherein each saidport circuit is structured to use the destination address tag of an FCALnet OPN primitive as a routing address and search key to search saidrouting table of destination addresses and structured to establish andupdate the contents of said routing table by learning the locations ofdestination nodes by watching traffic passing through said port, saidrouting table comprising a destination address and the ID of theparticular port to which data and primitives of an FCAL loop tenancy areto be sent to communicate with the NL node having that destinationaddress, said port circuits also structured to use the routinginformation derived from said search to search said scoreboard table andexchange messages with said remote port over a protocol bus to causesaid remote port to begin arbitrating and to select a particular datapath through said crossbar switch and to control switching by saidcrossbar switch to establish the selected data path.
 13. The apparatusof claim 8 wherein said crossbar switch is distributed and a portion ofit is integrated together with one or more of said port circuits on asingle integrated circuit die.
 14. A process for communicating databetween a source node and a destination node through one or more portsof a switch in a Fibre Channel protocol network, comprising: 1) in asource node, arbitrating for and winning control of a first FCAL net andtransmitting an OPN primitive thereon, said OPN primitive having adestination address of a destination node therein; 2) receiving andlatching at a first port of said switch said OPN primitive from saidsource node coupled to said first port by said first FCAL net; 3) usingthe destination address in said OPN primitive as a search key to searcha routing table to find the location of a destination node having saiddestination address or the ID of a port coupled by an FCAL net to saiddestination node, or both, and, if the destination node is coupled tosaid first port, passing said OPN primitive to said destination node viasaid first port via a local bypass data path coupling an input of thefirst port to an output of the first port coupled to the first FCAL net,but, if said destination node is coupled to a second port other thansaid first port, controlling a crossbar switch to establish a data pathbetween said first and second ports and determining if said second portis available, and, if so, sending said OPN primitive to said second portindicating traffic is waiting to be sent to said destination node andlatching said OPN in said second port; 4) in said second node,arbitrating for control of a second FCAL net coupled to said secondport; 5) when control of said second FCAL net is won following saidarbitration, forwarding said OPN to said destination node; 6) receivingan RRDY primitive or a CLS primitive from said destination node in saidsecond port and transmitting said primitive to said source node throughsaid crossbar switch, said first port and said first FCAL net; and 7)continuing to pass data frames and primitives between said source anddestination nodes, until a CLS primitive is transmitted by either saidsource node or said destination node, and then closing said data paththrough said crossbar switch and relinquishing control of said first andsecond FCAL nets.
 15. The process of claim 14 wherein step 1 furthercomprises the steps, of transmitting from said source node one or moreRRDY primitives immediately following transmission of said OPNprimitive, each RRDY primitive representing one frame buffer of creditwhich has been reserved in memory to store a frame of data transmittedback from said destination node to said source node, and wherein step 2further comprises the step of storing any RRDY primitives that weretransmitted following said OPN transmitted by said source node, andwherein step 5 further comprises the step of forwarding any RRDYprimitives received from said source node to said destination node, andwherein step 6 further comprises the steps of receiving any data framesoutput by said destination node in response to receipt of one or moreRRDY primitives from said source node and transmitting said data framesto said source node through said one or more ports of said switch. 16.The process of claim 14 wherein the step of determining if said secondport is available in step 5 comprises the steps of using said ID of theport coupled to said destination node determined by searching saidrouting table with the destination address in the OPN primitive receivedfrom the source address as a search key to search a scoreboard tablewhich stores port status information for every port and using saidstatus information to determine if the port coupled to said destinationnode is available where said status information indicates whether or notthe FCAL net on which the destination node resides is currently busy ina loop tenancy.
 17. A packet switching switch for coupling a pluralityof Fibre Channel NL nodes and/or Fibre Channel Arbitrated Loop network(herafter FCAL nets) together to all concurrent data exchanges between aplurality of pairs comprised of one NL node or FCAL net and one other NLnode or FCAL net, comprising: a crossbar switch; a protocol bus; aplurality of port circuits each having an input and an output forconnecting to an FCAL net coupled to one or more NL nodes and capable ofimplementing a Fibre Channel loop protocol, and each having a crossbarswitch port coupled to said crossbar switch; circuitry in each said portcircuit and coupled to said protocol bus for maintaining a scoreboardtable containing at least status information and a routing tablecontaining data mapping destination addresses of NL nodes to port IDs;and wherein said port circuits function to establish connections betweenthemselves by using destination addresses in OPN primitives receivedfrom source nodes to search said routing table to determine the ID of aremote port coupled to the destination node having the destinationaddress in the OPN and, using that port ID to search said scoreboardtable to determine status of the remote port, and then exchangingmessages with said remote port to cause it to arbitrate for and takecontrol of its local FCAL net and establish a data path via a particularchannel through said crossbar switch and use said channel to transmitprimitives and data frames between said source node and said destinationnode.
 18. The apparatus of claim 17 wherein each said port circuitincludes means for storing and updating the contents of said routingtable using a passive learning process by watching traffic through theport.
 19. The apparatus of claim 17 wherein each said port circuitincludes means for storing and updating the contents of said routingtable using an active discovery process of sending out inquiries to allnodes on the network.
 20. A packet switching switch for coupling aplurality of Fibre Channel NL nodes and/or Fibre Channel Arbitrated Loopnetwork (herafter FCAL nets) together to all concurrent data exchangesbetween a plurality of pairs comprised of one NL node or FCAL net andone other NL node or FCAL net, comprising: a crossbar switch; a protocolbus; a plurality of port circuits each having an input and an output forconnecting to an FCAL net coupled to one or more NL nodes and capable ofimplementing a Fibre Channel loop protocol, and each having a crossbarswitch port coupled to said crossbar switch; circuitry in each said portcircuit and coupled to said protocol bus for maintaining a routing tablecontaining data mapping destination addresses of NL nodes to port IDs;and wherein said port circuits function to establish connections betweenthemselves by latching any OPN and RRDY primitives received from sourcenodes and using destination addresses in OPN primitives received fromsaid source nodes to search said routing table to determine the ID of aremote port coupled to the destination node having the destinationaddress in the OPN, and then establishing a data path through saidcrossbar switch to said remote port and wherein each said port when itreceives an OPN and any following RRDYs from another port via saidcrossbar switch, functioning to latch said OPN and RRDY and arbitratefor and take control of its local FCAL net, and continuing to try totake conrol of its local FCAL net until control is achieved, and whencontrol is achieved, sending said OPN and any latched RRDYs to thedestination node, and thereafter using said data path through saidcrossbar switch to send primitives and data frames back and forthbetween said source and destination nodes until the transaction iscompleted without ever storing any data frames.
 21. The apparatus ofclaim 20 wherein each said port circuit includes means for storing andupdating the contents of said routing table using a passive learningprocess by watching traffic through the port.
 22. The apparatus of claim20 wherein each said port circuit includes means for storing andupdating the contents of said routing table using an active discoveryprocess of sending out inquiries to all nodes on the network.
 23. Anapparatus comprising: a plurality of port means for coupling to aplurality of FCAL nets, each FCAL net having one or more NL nodesthereon; and switching means coupled to said plurality of port means forcoupling selectable pairs of said port means together by separatechannels; and each said port means including means for receiving an OPNfrom a source node, determining which port the destination nodeidentified in the OPN is coupled to, and communicating with the portcoupled to the destination node to select one of said channels throughsaid switching means and establish said channel for communication ofprimitives and data frames between said source node and said destinationnode.
 24. The apparatus of claim 23 wherein each said port means has apriority level for access to busy destination nodes and wherein eachport means includes means for receiving and forwarding a fairness tokensuch that said fairness token circulates to all other said port means,and wherein each said port means includes means for using said fairnesstoken to increase the priority level of said port to a highest prioritylevel when said fairness token is in possession of said port.
 25. Aswitch for coupling a plurality of FCAL nets, comprising: plurality ofseparate backplane data paths; and plurality of port integrated circuitsselectably coupled together by said separate backplane data paths, eachintegrated circuit having integrated thereon a port circuit comprisingan FCAL net interface for coupling to one FCAL net and a portion of acrossbar switch coupled to said FCAL net interface and structured so asto capable of coupling said port to any selectable one of said separatebackplane data paths.
 26. The apparatus of claim 25 wherein each saidport integrated circuit has a priority level for access to busydestination nodes and wherein each port integrated circuit includesmeans for receiving and forwarding a fairness token such that saidfairness token circulates to all other said port integrated circuits,and wherein each said port integrated circuit includes means for usingsaid fairness token to increase the priority level of said portintegrated circuit to a highest priority level when said fairness tokenis in possession of said port integrated circuit.
 27. In a distributedsystem comprised of a plurality of nodes which are data consumers anddata senders coupled by a switch and a plurality of Fibre ChannelArbitrated Loops, a process for establishing one or more concurrent datatransfers between different pairs of nodes comprising the steps: foreach desired data transfer, sending to said switch from a source node arequest to start a loop tenancy with a destination node, said requestbeing in the form of a Fibre Channel Arbitrated Loop (hereafter FCAL)primitive and including a tag which identifies the destination node withwhich data is to be exchanged; using said request and tag in said switchto locate the FCAL upon which said destination node is resident,establishing a data transfer path between the FCAL upon which saidsource node is resident and the FCAL upon which said destination node isresident.
 28. In a distributed system comprised of a plurality of nodeswhich are data consumers and data senders coupled by a switch and aplurality of Fibre Channel Arbitrated Loops, a process for establishingone or more concurrent data transfers between different pairs of nodescomprising the steps: (1) for each desired data transfer, sending tosaid switch from a source node a request to start a loop tenancy with adestination node, said request being in the form of a Fibre ChannelArbitrated Loop (hereafter FCAL) primitive and including a tag whichidentifies the destination node with which data is to be exchanged; (2)receiving said request and tag in a switching circuit coupled to theFCAL upon which said source node is resident and using said tag todetermine from a table of nodes resident on said FCAL upon which saidsource node is resident if said destination node is resident thereon;(3) if said destination node is not resident on said FCAL upon whichsaid source node is resident, using a control channel on a multiplexedbus having data transfer channels and control channels to circulate saidrequest and tag to other switching circuits coupled each of said otherFCALs; (4) in each said other switching circuit, using said tag todetermine from a table of nodes resident on said FCAL to which saidswitching circuit is coupled if said destination node is resident on alocal FCAL coupled to said switching circuit; (5) generating in theswitching circuit coupled to the FCAL upon which said destination nodeis resident a grant message identifying the FCAL upon which saiddestination node is present by inclusion of the identification code ofthe switching circuit coupled to the FCAL on which said destination nodeis present in said grant message; (6) assiging one or more data transferchannels of said multiplexed bus to carry data of said desired datatransfer so as to establish a data transfer path between said switchingcircuit coupled to said FCAL upon which said source node is present andthe switching circuit coupled to said FCAL upon which said destinationnode is present to carry the data and FCAL primitives of the desiredFCAL protocol data transfer; and (7) using said switching circuitscoupled to said FCALs upon which said source and destination nodes areresident to couple an RRDY FCAL protocol primitive from said destinationnode to said source node after said data transfer path across saidmultiplexed bus has been established to trigger and control the flow ofdata, and placing the data of said desired data transfer in one or moreof said data transfer channels of said multiplexed bus to carry out saiddata transfer.
 29. The process of claim 28 wherein said multiplexed busis a time division multiplexed bus wherein some timeslots are dedicatedas control channels and other timeslots are dedicated to carrying data,and wherein step (3) is accomplished by placing said request and tag ina first one of said control channel timeslots known to all switchingcircuits, and wherein step (5) is accomplished by placing said grantmessage in a second one of said control channel timeslots known to allswitching circuits, and wherein step (6) is accomplished by assigningone of said timeslots dedicated to carrying data to carry the data ofthe desired data transfer by locating an unused data timeslot andbroadcasing the switching circuit identification code of the switchingcircuit coupled to the FCAL upon which said destination node is residentto all other switching circuits such that the switching circuit coupledto the FCAL upon which said source node is resident knows which timeslotto used to transfer further primitives and frame data to said switchingcircuit coupled to said FCAL upon which said destination node isresident, and wherein step (7) is accomplished by receiving data framesfrom said source node, marking the beginning and ending of every framewith a delimiter, breaking each frame into timeslot sized pieces, andplacing the pieces data of said desired data transfer into the timeslotassigned in step (6) for transfer to said switching circuit coupled tosaid FCAL upon which said destination node is resident.
 30. In adistributed system comprised of a plurality of nodes which are dataconsumers and data senders coupled by a switch and a plurality of FibreChannel Arbitrated Loops, a process for establishing one or moreconcurrent data transfers between different pairs of nodes comprisingthe steps: (1) for each desired data transfer, sending to said switchfrom a source node a request to start a loop tenancy with a destinationnode after arbitrating for and winning control of the FCAL upon whichsaid source node is resident, said request being in the form of FCAL OPNprimitive and including a tag which identifies the destination node withwhich data is to be exchanged; (2) receiving said request and tag in aswitching circuit coupled to the FCAL upon which said source node isresident and using said tag to determine from a table of nodes residenton said FCAL upon which said source node is resident if said destinationnode is also resident thereon; (3) if said destination node is notresident on said FCAL upon which said source node is resident, using acontrol channel on a multiplexed bus having data transfer channels andcontrol channels to broadcast a destination node location request andthe destination address of the destination node identified in said OPNprimitive, said request requesting other switching circuits coupled eachof said other FCALs to search their tables to determine which of them iscoupled to the FCAL to which the destination node is connected; (4) ineach said other switching circuit, using said destination address as asearch key to search a table of addresses of nodes resident on said FCALto which said other switching circuit is coupled (5) if said destinationnode is resident on a local FCAL coupled to said switching circuit andis available for a transaction, generating a grant message in theswitching circuit coupled to the FCAL upon which said destination nodeis resident, said grant message identifying the FCAL upon which saiddestination node is present by inclusion of the identification code ofthe switching circuit coupled to the FCAL on which said destination nodeis present in said grant message; (6) establishing a data transfer pathbetween said switching circuit coupled to said FCAL upon which saidsource node is present and the switching circuit coupled to said FCALupon which said destination node is present to carry the data and FCALprimitives of the desired FCAL protocol data transfer; and (7) usingsaid switching circuits coupled to said FCALs upon which said source anddestination nodes are resident and the data path established in step 6to couple an RRDY FCAL protocol primitive from said destination node tosaid source node after said data transfer path has been established totrigger and control the flow of data, and transferring data over saiddata transfer path between said source and destination nodes.
 31. Theprocess of claim 30 wherein step 6 is accomplished by receiving thegrant message at said switching circuit coupled to said FCAL upon whichsaid source node is resident and generating suitable steering signals toa crossbar switch which functions to selectively couple all switchingcircuits to each other under the influence of steering signals receivedfrom said switching circuits, said steering signals received from saidswitching circuit coupled to said FCAL upon which said source node isresident causing said crossbar switch to establish a data pathconnection between said switching circuit coupled to said FCAL uponwhich said source node is resident and said switching circuit coupled tothe FCAL on which the destination node is resident.
 32. A switch forselectively, concurrently coupling the nodes on a plurality of a FibreChannel Arbitrated Loops together comprising: a multiplexed bus; aplurality of switch control means, each for coupling to one local FCALnetwork, and each coupled to said multiplexed bus, each for receivingOPN primitives from a source node on the local FCAL network to which itis coupled and determining if the destination node identified in saidOPN is on said local FCAL network, and, if so, routing the OPN to thedestination node via a local traffic data path in said switch controlmeans and the local FCAL network, but if the destination node is not onsaid local FCAL, transmitting a destination node location request andthe destination address of the destination node identified in said OPNprimitive to all said other switch control means via a channel on saidmultiplexed bus, and for scanning messages received back from said otherswitch control means to determine which switch control means and localFCAL network to which flow control primitives and data are to betransferred to accomplish the desired data transfer, and when a grantmessage is received from another switch control means indicating saidother switch control means is the place to send flow control primitivesand data and indicating that the FCAL to which said other switch controlmeans is coupled is available for a loop tenancy, for cooperating withsaid other switch control means to establish a data transfer path acrosssaid multiplexed bus, and using said data transfer path to transfer flowcontrol primitives between said source and destination nodes, and eachswitch control means also for receiving destination node locationrequests and destination addresses, and using said destination addressto search a table of destination addresses of nodes on the local FCAL ofsaid switch control means to determine if the destination nodeidentified in said destination node location request is on the localFCAL to which said switch control means is coupled, and, if so, forarbitrating for control of said local FCAL and, when control is won,sending a grant message back to said switch control means whichinitiated destination node location request identifying the switchcontrol means coupled to the local FCAL on which said destination nodeis resident and for sending one or more messages to said switch controlmeans which initiated said destination node location request forpurposes of controlling establishment of said data transfer path.
 33. Aswitch for selectively coupling pairs of nodes on a plurality of FCALnets to each other for one or more simultaneous conversations,comprising: data path circuitry having a plurality of parallelpoint-to-point transmit/receive channels and multiplexing circuitry toselectively couple individual ones of said transmit/receive channels totransmit-receive terminals; a protocol bus; a plurality of ports eachport for interfacing a local FCAL net to said data path circuitry, eachport coupled to said protocol bus and comprising: aserializer/deserializer circuit having terminals for coupling to an FCALnet and having demultiplexing circuitry to convert incomingdifferential, encoded serial data from said FCAL net to multi-bitcharacters and having clock recovery circuitry to recover a receiveclock therefrom and having multiplexing circuitry for receiving aplurality of multi-bit characters and converting them into adifferential, encoded serial data stream; an elastic buffer to receivemulti-bit characters from and send multi-bit characters to saidserializer/deserializer circuit and absorb the differences in transmitand receive rate; a routing table memory storing mappings betweendestination node addresses and port IDs of the port coupled to eachdestination node; scoreboard means for storing status information; oneor more state machine means coupled to said elastic buffer, said routingtable memory, said scoreboard means and said protocol bus and at leastto said transmit-receive terminals of said data path circuitry for usingsaid routing table memory contents for determining whether destinationnodes identified in OPN primitives received from source nodes on saidlocal FCAL net are local or remote and using the data content of saidscoreboard means to determine if connection to a destination portcoupled to said destination node is permissible and transmitting aconnection request on said protocol bus to said destination portrequesting a connection thereto and using the contents of a connectresponse frame output by said state machine means of said destinationport on said protocol bus for controlling said data path circuitry toimplement full duplex connections between said local source and remotedestination nodes or to implement full or half duplex connections toremote destination nodes, and for implementing logic and controlfunctions to control said data path circuitry to carry out a dualsimplex connections so as to send data and primitives from a source nodeon said local FCAL net to a remote dual simplex destination node coupledto some other port which is a source node in another half duplex looptenancy and receive buffer credit flow control primitives back from saiddual simplex destination node and to respond to each said buffer creditflow control primitive by transmitting a frame of data to said dualsimplex destination node, said state machine means also structured forsending flow control primitives to a third port for transmission to athird node and receive OPN and CLS primitives and data frames from saidthird node when said third port has established a dual simplexconnection to said port through said data path circuitry, and transmitdata frames and CLS primitives received from said third node to saidsource node if and only if said data frames are directed to said sourcenode as indicated by said OPN received from said third node.
 34. Theapparatus of claim 33 wherein said state machine means includes meansfor receiving configuration commands at a management interface portcausing said state machine means to control said data path circuitry toassign and couple one or more transmit-receive channels to acorresponding number of one or more receiving ports on a continuingbasis until no other transmit-receive channels are available to serviceconversations between pairs of ports and then release one or more ofsaid assigned channels to service a conversation between pairs of portswhich currently do not have an assigned channel.
 35. The apparatus ofclaim 33 wherein said state machine means in every port includes meansfor receiving configuration commands at a management interface portcausing said state machine means to control said data path circuitry toassign and couple a transmit-receive channel to said port if it iscoupled to either the source node or destination node of a desired looptenancy defined by an initial OPN primitive transmitted by said sourcenode and to maintain said connection until either said loop tenancy isterminated, and, thereafter, to control said data path circuitry torelease said channel and to control said state machine to publish aresponder channel idle data frame on said protocol bus.
 36. Theapparatus of claim 33 wherein each said state machine means in everyport includes means for circulating a fairness token to guarantee remoteaccess fairness to ports by rotating a fairness token that guaranteeshigh priority access among all ports.
 37. The apparatus of claim 33wherein each said state machine means in every port includes means forescalating a no-priority request to a low-priority request based uponthe number of consecutive denials of access issued by a state machinemeans of a remote port.
 38. The apparatus of claim 36 wherein each saidstate machine means in every port includes means for escalating ano-priority request to a low-priority request based upon the number ofconsecutive denials of access issued by a state machine means of aremote port.
 39. The apparatus of claim 33 wherein every state machinemeans further comprises means to originate and receive broadcastsequences.
 40. A bufferless switch for coupling to a plurality of FCALnets and having a crossbar switch and FCAL loop interface port circuitsstructured to use the OPN and RRDY primitives of the FCAL protocol forhold back flow control to eliminate the need for a buffer and said portsand crossbar switch structured to provide multiple simultaneous looptenancies during switch mode or to couple all FCAL nets together as onebig FCAL net during hub mode, and wherein said switch can be switchedback and forth between hub mode and switch mode by going through anintermediate locked mode.
 41. A process for carrying out dual simplexcommunications in a FCAL switch coupling a plurality of FCAL nets,comprising: receiving and latching an OPN and at least the source nodeaddress thereof identifying the address of said source node andreceiving any following buffer credit RRDY primitives and either storingsaid RRDYs or counting the number thereof at a source port from a sourcenode on the local FCAL net coupled to said source port, said OPNdirected to a remote destination node on the FCAL net coupled to aremote port; establishing a front channel connection through a backplanebetween said source port and said destination port; converting said OPNto half duplex and transmitting said half duplex OPN to said destinationOPN over said front channel; receiving any RRDY primitives from saiddestination node and transmitting them to said source node via a backchannel; receiving a connection request message from a third portcoupled to a third node and granting said connection request by aconnection grant message to said third port coupled to said third node,said connection grant message naming a backplane channel to use for aback channel connection between said source node and said third node;establishing a backchannel connection between said source port and saidthird port through said backplane channel named in said connection grantmessage; transmitting an OPN directed to said source node from saidthird node to said source port; comparing the destination node addressin said OPN from said third port to the node address of the OPNoriginally received by said source port, and, if the addresses match,quashing the OPN from said third port and either transmitting one storedRRDY received from said source node to said third node or transmitting amessage which causes said third port to generate an RRDY and transmit itto said third node; transmitting one frame of data from said third nodeto said source node via said back channel connection; and continuing tointercept RRDY primitives transmitted by said source node and send themto said third node and to receive frames of data transmitted in responseto receipt at said third node of said RRDYs and transmit said frames tosaid source node via said back channel connection and mix said frames ofdata on said back channel connection with RRDY primitives received fromsaid destination node.
 42. A process for dual simplex communicationusing a switch coupling a plurality of FCAL nets, comprising:establishing a front channel half duplex data path between a source nodeand destination node on different FCAL nets and stripping and storing orcounting any buffer credit RRDY primitives output by said source nodeand not transmitting them to said destination node; establishing a backchannel data path between a third node and said source node but nottransmitting to said source node any OPN primitive emitted by said thirdnode, and transmitting a number of RRDYs either equal to the number ofRRDYs output by said source node or the number of RRDYs needed by saidthird node to send all the data it has to said source node beforeclosing said back channel connection, transmission of said RRDYs beingone at a time; and receiving any RRDYs transmitted by said destinationnode and mixing them in with data frames and/or primitives transmittedon said back channel by said third node so as to exercise flow controlon transmissions of data frames from said source node to saiddestination node.
 43. A process for dual simplex communication using aswitch coupling a plurality of FCAL nets, comprising: 1) receiving afull duplex OPN(Dest, Src) from a source node designating a destinationnode address, Dest, and a source node address, Src, and converting thefull duplex format OPN(Dest, Src) to a simplex or half duplex formatOPN(Dest, Dest) and establishing a front channel connection to saiddestination node; 2) storing the full duplex OPN including at least thesource node address in a source port of said switch coupled to saidsource node; 3) stripping any buffer credit RRDYs output by said sourcenode and not transmitting said RRDYs output by said source node to saiddestination node of said front channel connection, but either storing orcounting said RRDYs output by said source node and making them availableto a third node to send data to said source node either by transmittingstored RRDYs output by said source node one at a time to said third nodeor by conveying the number of RRDYs output by said source node to athird port coupled to said third node and synthesizing in said thirdnode a number of RRDYs equal to the value of said count and sending saidRRDYs to said third node one at a time; 4) updating the status data in ascoreboard memory of each port to indicate which busy ports are coupledto source nodes of front channel connections and thus are available toreceive frames in dual simplex communication from third nodes; 5)establishing a back channel connection from said third node to saidsource node of said front channel connection and transmitting an OPNfrom said third node to said source port, and using said stored sourcenode address from said OPN transmitted by said source node and stored insaid source port as an access key to compare to the destination nodeaddress of said OPN transmitted to said source port by said third nodevia said back channel connection and allowing dual simplex communicationto proceed on said back channel if said source node address matches saiddestination node address of said OPN from said third node, but if thereis no match, blocking dual simplex communication with this particularthird node and closing said back channel connection; 6) deleting saidOPN from said third node in said source port and never transmitting itto said source node if the destination address of the OPN from saidthird node matches the stored source address of the stored OPN from saidsource node; and 7) receiving data frames transmitted by said third nodeat said source port with one data frame transmitted in response to eachRRDY received by said third node, and mixing said data frames in withRRDYs received from said destination node via said front channelconnection and transmitting the combined data frames and RRDYs to saidsource node via said back channel connection.
 44. A switch apparatuscomprising a plurality of integrated circuits each comprising a slice ofan FCAL switch and coupled together to form said FCAL switch, eachintegrated circuit having a plurality of port circuits, each portcircuit having substantially less buffer memory to store data than priorart FL ports in prior art fabric switches, each said integrated circuitalso comprising a portion of a distributed, scaleable crossbar switchsuch that any number of said integrated circuit slices can be coupledtogether to form an FCAL switch which has as many ports as are necessaryfor the size of network in which the switch is to be used, limited onlyby the available FCAL address space of 128 nodes.
 45. The apparatus ofclaim 44 wherein said crossbar switch is structured such that when allof said integrated circuits are coupled together, a complete crossbarswitch is formed, said integrated circuit or circuits including a lookuptable and circuitry to use the destination address of an OPN primitivearriving from a local FCAL net coupled to a port circuit to access saidlookup table to determine the FCAL net and port coupled to a destinationnode having that destination address and to determine whether to makeconnections so as to forward the OPN back onto the local FCAL net fromwhich the OPN arrived and keep any subsequent data frames and primitiveson the local FCAL net or to forward the OPN to another port on the sameintegrated circuit for coupling onto its FCAL net and transmitsubsequent data frames and primitives between the FCAL net coupled tothe port which received the OPN and the FCAL net coupled to another porton the same integrated circuit having the destination node identified insaid OPN or to make connections through said crossbar switch to forwardsaid open to a port on another integrated circuit and an FCAL netcoupled to said port and said destination node and to transfer anysubsequent data frames and primitives between said FCAL nets coupled todifferent integrated circuits through said crossbar switch and the portscoupled to said FCAL nets.
 46. The switch of claim 44 wherein said portcircuits and crossbar switch circuitry are structured so as to becapable of cooperating to implement dual simplex communications.
 47. Theswitch of claim 44 wherein said switch contains a memory for storingconfiguration data and wherein said configuration data containsprogrammable data indicating whether or not dual simplex communicationis allowed, and wherein said port circuits and crossbar switch circuitryare structured so as to be capable of cooperating to implement dualsimplex communications when said configuration data indicates dualsimplex communication is allowed.
 48. A process for insuring fairness isachieved in an switch having a plurality of switch ports, each coupledto a an FCAL net having one or more nodes coupled thereto, comprising:assigning each switch port a priority level for use in arbitrationbetween conflicting connect requests from multiple switch ports to thesame destination port; circulating a fairness token among said switchports so that each switch port receives said fairness token, holds itfor a time and then forwards it to another switch port; assigning aswitch port that has said fairness token in its possession the highestof said priority levels.
 49. The process of claim 48 further comprisingthe steps of: keeping a camp list for busy ports which have receivedmultiple connect requests from different switch ports; when a switchport in possession of said fairness token makes a connect request to aswitch port with a camp list which is not full, adding the ID of saidswitch port having said fairness token to said camp list; when access tosaid switch port with the camp list is granted to the switch port inpossession of the fairness token, forwarding said fairness token toanother switch port.
 50. An FCAL buffered switch for coupling to aplurality of FCAL nets and having a crossbar switch and FCAL loopinterface port circuits structured to use the destination address of theOPN to find the destination node and destination port and includingswitching circuitry coupled to said crossbar switch by by multiple pathsincluding at least a straight through path and paths through one or morebuffer memorys to exchange data frames and FCAL primitives with one ormore destination nodes and structured to use one or more buffer memoriesto store data output by a source node destined for a destination nodewhich has a busy status at the time the data is output by said sourcenode, said ports and crossbar switch structured to provide multiplesimultaneous loop tenancies.
 51. An FCAL buffered switch for coupling toa plurality of FCAL nets and having a crossbar switch and FCAL loopinterface port circuits structured to use the destination address of theOPN to find the destination node and destination port and includingswitching circuitry coupled to said crossbar switch by by multiple pathsincluding at least a straight through path and paths through one or morebuffer memorys to exchange data frames and FCAL primitives with one ormore destination nodes and structured to use one or more buffer memoriesto store data output by a source node destined for a destination nodewhich has a busy status at the time the data is output by said sourcenode, said ports and crossbar switch structured to provide multiplesimultaneous loop tenancies and said ports and crossbar switch alsostructured to provide dual simplex communication across said switch. 52.An FCAL buffered switch for coupling to a plurality of FCAL nets andhaving a crossbar switch and FCAL loop interface port circuitsstructured to use the destination address of the OPN to find thedestination node and destination port and including switching circuitrycoupled to said crossbar switch by by multiple paths including at leasta straight through path and paths through one or more buffer memorys toexchange data frames and FCAL primitives with one or more destinationnodes and structured to use one or more buffer memories to store dataoutput by a source node destined for a destination node which has a busystatus at the time the data is output by said source node, said portsand crossbar switch structured to provide multiple simultaneous looptenancies and said ports and crossbar switch also structured to providedual simplex communication across said switch, each said port includinga local bypass data path which can be switched to keep primitives anddata frames generated on a local FCAL net coupled to said port routed soas to stay on said local FCAL net such that local loop tenancies on eachlocal FCAL net coupled to a port which do not involve transmission ofdata or primitives from one port to another can occur simultaneously,and each port including local learning memory means for storing arouting table and scoreboard table, and functioning to fully cache the8-bit addresses of all nodes on each said local FCAL net coupled to aport along with the ID of the port coupled to each node in said routingtable and the busy, available or no privilege status of each other portand whether each busy port is available for dual simplex communicationsin said scoreboard table, each port structured to write the contents ofeach said routing table and scoreboard table by learning eitherfrom,watching local traffic in said port bound for local destinationnodes or remote traffic leaving said port and crossing said crossbarswitch bound for remote destination nodes or by conducting an activediscovery process, each port structured to write the contents of saidscoreboard table by learning from watching messages posted on a protocolbus, each said port structured to use destination addresses in OPNprimitives transmitted by nodes on said local FCAL net as search keys tosearch said routing table to determine whether the destination node islocal or remote and to which port it is connected and, if local, toswitch said local bypass to keep the loop tenancy confined to said localFCAL net, or, if remote, to generate a connection request message tosaid port coupled to said destination node, each said port beingstructured to receive such connection requests and respond byarbitrating for control of its local FCAL net and pick a data paththrough said crossbar switch and generate a reply message naming thedata path through said crossbar switch to use in exchanging data framesand primitives, each said port also being structured to respond toreceipt or generation of said reply message by generating commands toestablish said data path named in said reply message through saidcrossbar switch, each said port also structured to circulate a fairnesstoken on a fairness token bus coupled to all such ports and to used saidfairness token to increase its priority level of access privilege tobusy nodes when the fairness token is in possession of the port so thatno port can be starved, and then to circulate said fairness token to aneighboring port such that all ports eventually get said fairness token.